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FOREWORD 


The  test  and  evaluation  of  models  developed  in  Phase  II  of  the 
Design  of  Training  Systems  (DOTS)  project  was  conducted  under  Advanced 
Development  Objective  ZPN07  (formerly  P43-03X),  entitled  "Education  and 
Training  Development." 

This  test  and  evaluation  was  possible  only  through  the  cooperation 
of  several  individuals  and  organizations.  The  User  T&E  group  was  com¬ 
prised  of  CDR  Jack  Davis  and  Mr.  David  Thomas  (CNTT) ,  LCDR  Thomas  Ferrier 
(COMTRAPAC) ,  Kr.  Edward  Scheye  (CNET) ,  LCDR  Bob  Biersner  (CNETS) ,  and 
LT  Ross  Brooks  (COMTRALANT) .  These  individuals  in  particular  and  their 
commands  in  general  bore  the  brunt  of  providing  the  necessary  information 
to  assess  operational  feasibility.  The  Training  Analysis  and  Evaluation 
Group  (TAEG)  and  the  contractor,  IBM  Federal  Systems  Division,  were  exceed¬ 
ingly  cooperative  throughout  the  course  of  the  T&E.  Mr.  William  Lindahl 
(TAEG)  performed  an  indispensable  role  and  was  supportive  throughout  the 
T&E.  Dr.  Raymond  Willis  (University  of  Minnesota)  provided  key  insights 
to  the  technical  evaluation  of  the  models  and  suggestions  for  alternative 
statistical  approaches  (Appendix  G) . 


J.  J.  CLARKIN 
Commanding  Officer 


SUMMARY 


PROBLEM 

An  Advanced  Development  Project  titled,  Design  of  Training  Systems  (DOTS), 
has  attempted  to  demonstrate  the  feasibility  of  applying  new  decision  making 
technologies,  based  upon  the  behavioral  and  management  sciences,  to  the 
management  of  tiaining  in  the  Navy.  Several  prototype  applications  were 
developed  using  mathematical  modeling  approaches.  It  was  necessary  to 
independently  ascertain  whether  feasibility  had  in  fact  been  demonstrated 
through  the  prototypes  thus  far  developed. 

OBJECTIVE 

A  test  and  evaluation  was  undertaken  in  order  to  determine  the  technical, 
operational,  and  financial  feasibility  of  the  prototypes  as  well  at  of  the 
broader  applications  of  the  technologies  in  question  throughout  the  Navy 
Education  ana  Training  community. 


APPROACH 

Thorough  documentation  of  the  preliminary  system  analysis,  model  develop¬ 
ment,  and  model  validation  was  provided  for  technical  review.  The  model 
formulations  were  analyzed  for  theoretical  soundness,  sensitivity  to  the 
problem,  and  validity  of  problem  representation.  An  operational  test 
and  evaluation  was  conducted  to  include  the  following:  (1)  determination 
of  data  availability,  validity  and  reliability,  (2)  operation  of  the  models 
using  real  (current)  data,  (3)  "hands-on"  user  evaluation  of  the  models 
by  representives  of  various  Naval  Education  and  Training  Command  functions 
and  (4)  position  papers  from  prospective  user  commands  regarding  implemen¬ 
tation  of  the  prototypes.  A  determination  of  financial  feasibility  was 
approached  through  analysis  of  the  component  cost  of  model  usage,  projection 
of  model  utilization  and  gross  estimators  of  benefits  stemming  from  the 
models . 


CONCLUSIONS 

The  models  were  determined  to  be  technically  sound  overall.-  In  certain 
instances  alternative  approaches  might  srrike  a  more  effective  balance 
between  detail  of  information,  and  practicality  of  operation.  The  validity 
of  the  models  was  demonstrated  with  regard  to  software  coding.  However, 
a  comprehensive  validation  must  still  be  accomplished  through  field  testing 
under  operational  conditions  prior  to  implementing  any  of  these  prototypes. 
The  model  prototypes  were  judged  to  be  operationally  and  financially  feasible 
for  application  at  the  Fleet  Training  Command  level. 


v 


The  higher  order  feasibility  of  applying  the  technologies  represented 
by  the  prototypes,  throughout  the  Navy  education  and  training  community 
was  positively  indicated  to  the  extent  possible  given  financial  and 
temporal  constraints  of  the  model  development  project. 


RECOMMENDATIONS 

Data  collection  and  review  for  field  testing  of  the  System  Capabili¬ 
ties/Requirements  and  Resources  (SCRR)  and  Training  Process  Flow  (TPF) 
models  at  the  Fleet  Training  Commands  should  be  initiated.  These  models 
should  be  modified  as  necessary  to  operate  at  the  training  command  level. 
The  ETE  model  should  be  given  greater  exposure  to  ILS  developers  in  the 
training  community  and  appropriate  field  test  for  chat  model  should  be 
arranged  (pp.  57ff). 

Since  it  appears  that  any  one  model  or  system  of  models  would  be 
insufficient  to  comprehensively  address  present  and  future  Navy  training 
management  problems,  a  continuing  capability  to  apply  the  kinds  of 
technologies  reflected  by  the  DOTS  prototypes  should  reside  within  CNET 
(pp.  57ff). 
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INTRODUCTION 


PROBLEM 

The  Design  of  Training  Systems  (DOTS)  project  has  attempted  to  demon¬ 
strate  the  feasibility  of  introducing  new  technologies  of  education, 
psychology,  management,  and  operations  research  to  the  management  of  Navy 
training.  Several  prototype  applications  were  developed  using  mathematical 
modeling  approaches.  Three  of  these  prototypes  were  selected  for  further 
development.  The  technical,  operational,  and  economic  feasibility  of 
these  models  must  be  determined. 

OBJECTIVE 

The  primary  objective  of  the  test  and  evaluation  reported  herein  was 
to  determine  the  technical,  operational,  and  financial  feasibility  of  the 
selected  prototypes.  A  second  objective  was  to  determine  the  broader 
applications  of  the  technologies  in  question  throughout  the  Navy  education 
and  training  community. 

BACKGROUND 

The  DOTS  project  is  Part  01A  of  the  Technical  Development  Plan  (TDP) 
for  Advanced  Development  Objective  (ADO)  ZPNO/  (Education  and  Training 
Development).  The  Principal  Development  Agency  for  this  ADO  is  the  Navy 
Personnel  Research  and  Development  Center  (NAVPERSRANDCEN) .  The  DOTS 
project  has  been  managed  by  the  Training  Analysis  and  Evaluation  Croup 
(TAEG) ,  and  the  principal  model  development  has  been  performed  by  a  con¬ 
tractor,  IBM  Federal  Systems  Division. 

The  DOTS  project  took  a  three-phased  form.  Phase  I  consisted  of  a 
systems  analysis  of  the  Naval  Education  and  Training  System.  This  phase 
resulted  in  (1)  a  descriptive  functional  model  of  that  system,  (2)  a  set 
of  strategic  assumptions  concerning  that  system  in  the  1980  time  frame, 
and  (3)  the  identification  of  a  number  of  candidate  computer-based  mathema¬ 
tical  model  types  which  could  support  Navy  training  management. 

Phase  II  consisted  of  the  selection,  development,  and  validation  of 
three  of  these  model  types,  as  well  as  development  of  a  support  data  base. 
The  three  models  selected  were  (1)  the  System  Capabilities/Requirements 
and  Resources  (SCRR)  Model,  (2)  the  Training  Process  Flow  (TPF)  Model, 
and  (3)  the  Educational  Technology  Evaluation  (ETE)  Model.  These  models 
were  subsequently  formulated  and  validated  by  the  contractor. 

Phase  III  consisted  of  further  validation  and  verification  of  the 
models.  The  DOTS  TDP  explicitly  called  for  an  independent  test  and 
evaluation  of  the  Phase  II  models  (see  Appendix  A).  NAVPERSRANDCEN  was 
charged  with  the  responsibility  of  conducting  this  T&E,  with  assistance 
to  be  provided  by  representatives  from  the  Chief  of  Naval  Education 
and  Training  (CNET)  and  its  functional  commands  (see  Appendix  B) . 

For  further  background  information  on  the  DOTS  project,  see  TAEG  Reports 
11-1,  Summary  Report,  of  December  1973  and  21-2,  Volume  I,  Phase  11 
Overview,  of  December  1974. 


ON-SITE  USER  TES1  AND  EVALUATION 


General 


The  on-sitc  test  and  evaluation  (T&E)  was  conducted  at  the  Fleet 
Training  Center  ( FEET RAC EN ) ,  Norfolk,  Virginia  during  the  period  from 
23  to  27  June  1975.  The  initial  T&E  plan  formulated  and  iistributed  by 
NAVPERSRANDCEN  (Appendix  C)  established  a  User  Evaluation  Team  comprised 
of  representatives  of  CNET  and  its  functional  commands.  The  primary 
responsibility  of  the  User  Evaluation  Team  was  to  develop  a  comprehensive 
statement  of  evaluation  focussing  on  the  operational  aspects  of  the 
Phase  II  models.  NAVPERSRANDCEN  was  to  coordinate  this  user  evaluation 
and  take  primary  responsibility  for  technical  and  economic  assessment 
of  the  models. 


Technical  Feasib i llcy  Assessment 

A  thorough  technical  review  was  made  of  the  documentation  on  the 
preliminary  system  analysis  (Phase  1)  and  model  application,  selection, 
formulation,  and  validation  (Phase  11).  The  model  formulations  were 
analyzed  for  theoretical  soundness,  sensitivity  to  the  problem,  and 
validity  of  problem  presentation.  Consideration  was  given  to  (1)  the 
relationship  between  the  kinds  of  information  produced  by  the  model  and 
the  information  needed  by  management  in  the  real  world,  (2)  the  manner 
in  which  the  information  is  developed  within  the  models,  and  (3)  the 
data  upon  which  modeL  values  are  based.  Although  NAVPERSRANDCEN  was 
primarily  responsible  for  this  review,  inputs  were  provided  by  members 
of  the  User  Evaluation  Team.  Results  are  presented  in  the  following 
section. 

Operational  Feasibility  Assessment 
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On-site  User  Evaluation 


A  model  evaluation  framework  was  formulated  and  distributed  by 
NAVPERSRANDCEN  to  User  Evaluation  Team  members  (Appendix  D) .  The  assess¬ 
ment  of  the  operational  feasibility  of  the  Phase  II  models  was  to  he 
approached  by  holding  discussions  on  the  following  five  major  areas: 

(1)  potential  model  contributions,  (2)  ease  and  practicality  of  use, 

(3)  data  requirements,  (4)  organizational  implementation,  and  (5)  user 
investment.  These  discussions  were  held  during  the  first  3  days  of  the 
User  T&E.  During  the  first  session,  i.e.,  on  potential  model  contributions, 
team  members  were  provided  with  a  number  of  command  level  problem  areas 
identified  by  FLETRACEN  personnel  and  a  set  of  criteria  with  which  to 
evaluate  these  problem  areas  (Appendix  E) .  During  the  last  2  days  of  the 
User  T&E,  team  members  developed  a  comprehensive  evaluation  statement 
covering  the  results  of  the  discussions.  This  statement  is  presented  in 
a  later  section.  Although  the  text  of  this  statement  was  reviewed  by 
all  representatives,  divergent  opinions  in  some  areas  are  still  evident. 
Along  with  the  evaluation  statement .  team  members  developed  a  number  of 
conclusions  and  recommended  that  CNET  function  representatives  meet  again 
to  finalize  the  T&E. 
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User  T&K  Followup  Meeting 
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In  response  to  the  User  T&E  Team  recommendation  for  a  second  and  final 
meeting,  the  representatives  were  again  contacted  (via  phone)  in  August . 
Each  of  the  representatives  had  an  opportunity  to  present  the  DOTS  models 
and  user  evaluation  to  their  respective  commands.  It  was  hoped  that 
through  this  process  a  broader  feedback  from  potential  DOTS  usei s  could  be 
fed  into  the  T&E.  It  was  also  thought  that,  given  general  acceptability 
of  the  models,  the  potential  user  organizations  might  provide  an  explicit 
account  of  their  position  vis  a  vis  the  models.  This  would,  of  course, 
greatly  facilitate  follow-on  development  or  complementation  of  the  DOTS 
products . 


The  need  for  a  second  user  meeting  seemed  far  less  pressing  after 
these  discussions.  For  the  most  part,  the  positions  of  the  various 
representatives  had  crystalized  as  a  result  of  discussions  within  their 
commands.  The  representatvies  from  CNETS  and  CNET  had  much  less  of  a 
basis  lor  providing  additional  feedback  since  their  organizations  would 
not  be  immediate  consumers  of  the  models.  The  other  representatives 
either  were  in  the  process  of  completing  written  command  position  statements 
or  had  already  done  so.  It  appeared  that  an  additional  meeting  with  the 
former  two  representatives  was  unnecessary;  whereas,  individual  meeting 
with  the  latter  representatives  promised  to  be  more  fruitful*  than  another 
joint  meeting. 

Consequently,  the  CNET  and  CNETS  representatives  torwarded  statements 
of  position  along  with  copies  of  the  User  T&F  report  with  lurther  comments 
and  editorial  changes  (see  Appendix  F) .  Separate  face-to-face  meetings 
were  arranged  with  the  TRAPAC,  TRALANT,  and  CNTT  Representatives.  As  a 
result  of  these  meetings,  briefings  on  the  DOTS  T&E  were  provided  to  RADM 
Hill,  RADM  Gibbons,  and  RADM  Sackett,  respectively.  (RADM  Sackett  and  his 
staff  were  also  given  a  demonstration  of  the  DOTS  models  by  IBM).  In 
addition,  written  position  papers  concerning  DOTS  were  obtained  from  each 
of  the  commands  (See  Appendix  F). 

Fleet  Training  Command  Evaluation 
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A  final  phase  of  the  user  followup  took  place  at  COMTRAPAC  in  September. 
Throughout  the  T&E  there  had  been  a  dearth  of  feedback  as  to  FTC,  Norfolk 
reactions  to  the  specif ic  utility  of  the  various  model  information  products, 
though  some  general  reactions  had  been  provided  by  the  FTC  DOT  and  the 
TRALANT  T&E  representative.  It  was  determined  that  while  the  FTC  application 
was  not  a  field  test  per  se ,  the  level  of  model  development  and  opportunity 
for  application  was  such  that  specific  examples  of  model  utility  using 
actual  (if  historical)  problems,  along  with  FTC  management  reactions, 
should  have  been  produced  in  the  course  of  DOTS  Phase  111.  The  potential 
model  contribution  questions  (Appendix  E)  were  an  attempt  to  approach 


*"Kruitful"  in  the  sense  that,  these  being  the  most  likely  immediate 
users  of  the  models,  prospects  for  specific  implementation  could  be  more 
easily  and  comprehensively  auuiesseu.  Furthermore,  there  was  r«n  require¬ 
ment  for  a  team  concensus  of  opinion. 


vs 
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this  level 
necessary, 
models  and 
result  of  t 


of  specificity.  But  a  moie  direct  assessment  still  seemed 
Furthermore,  some  quest i mis  concerning  ease  ot  use  of  the 
supporting  documentation  still  remained  unanswered  as  u 
he  on-site  User  T&K. 


Consequently,  an  additional  user  test  and  evaluation  was  arranged 
at  COMTRAPAC .  This  was  a  small  scale  effort  direct ly  Involving  only 
the  COMTRAPAC  and  COM i RALAN  T  T&F  representatives  as  well  as  some  local 
support  personnel.  These  two  training  managers  were  provided  with  a 
pool  of  sample  model  problems  which  they  could  modify  or  draw  from  as 
they  saw  fit  in  order  to  properly  assess  the  value  ol  model  outputs. 
Since  there  were  to  be  no  IBM,  TAEG  or  NPRDC  personnel  present  Outing 
the  assessment,  it  was  also  considered  a  good  test  of  the  models  ease 
of  use,  given  their  current  level  of  conversational  programming  and 
supporting  documentation.  In  sum,  it  was  thought  that  this  assessment 
would  provide  an  indicator  of  how  close  the  models  were  to  being  imple- 
mentable  techniques  at  a  different  training  complex  which  possesses 
similar  environment  and  composition  to  the  development  site  at  Norlolk. 


Two  problems  were  immediately  encountered  in  this  assessment.  The 
first  was  that  of  equipment  compatibility.  The  training  management  ceutet 
at  COMTRAPAC  presently  has  a  communicating  terminal  of  a  different  typo 
than  that  used  at  Norfolk.  Certain  settings  and  adjustments  were  required 
through  the  time  sharing  vendor  before  the  equipment  could  bo  mode  to 
function  properly  in  response  to  the  time  sharing  system.  A  second  problem 
was  that  the  available  model  documentation  generally  proved  inadequate 
for  this  group  of  users.  The  main  problem  was  the  absence  of  information 
through  which  to  interpret  error  messages  and  to  rectify  incorrect  responses. 


This  situation  largely  prohibited  using  the  models  to  address  a  number 
of  real  training  management  problems  that  arose  during  the  course  of 
this  assessment.  For  example,  there  had  been  a  congressional ly  based 
inquiry  as  to  the  probable  impact  of  doubling  the  proportion  of  student 
AOB  to  instructors  and  supporting  staff.  This  could  have  been  a  fine 
application  of  DOTS  model  capability.** 


Adjustments  were  made 
period,  the  models  were  c 
Thus  it  has  been  ueiiujusii 
be  accessed  by  COMTRAPAC. 
provided  computer  tormina 
levels,  it  can  be  seen  th 
implementation  of  the  mod 
from  TRAPAC  school  admini 
techniques  to  their  probl 
interest  of  TRAPAC  in  the 


so  that  by  the  conclusion  of  the  assessment 
ompletely  functioning  on  COMTRAPAC's  equipment, 
ated  that  the  models  in  their  current  foim  can 
Furthermore,  since  CONTRAPAC  presently  has 
1  capability  to  both  seboolhouse  and  staff 
at  the  way  is  well  paved  for  field  testing  and 
els  3 1  that  command.  Also,  a  keen  interest 
strators  in  applications  of  management  science 
ems  was  reported.  This  complemented  the  general 
MIS  capabilities  of  the  DOTS  data  base  ana 


*lt  should  be  pointed  out  that  at  this  lime,  IBM  was  at  work  on  further 
user  documentation  that  would  offer  greater  support  of  this  type. 

**0f  course,  even  if  the  models  had  been  used,  the  answer  would  have  only 
been  a  sample  one  since  the  present  model  data  base  only  represents  tic, 
Norfolk. 


improvements  that  it  makes  on  present  NITRAS  capabilities.  For  example, 

N1TRAS  does  not  presently  contain  facilities  information.  Thus  TRAPAC 
is  unable  to  use  it  ot  address  facilities  plans  in  connection  with  the 
Program  Objectives  Memorandum  (POM)  process. 

Comments  on  User  T&E 

It  is  fairly  obvious  from  the  above  account  that  a  preponderance  of 
consideration  was  given  to  operational  feasibility  and  user  relevance 
throughout  the  T&E.  This  was  intentional  since  it  became  apparent,  after 
an  initial  review  of  DOTS,  that  organizational  implementation  was  the 
pivotal  issue  in  the  DOTS  feasibility  test.  Any  difficulties  in  overcoming 
technical  problems  in  modeling  this  particular  organizational  environment 
are  dwarfed  by  problems  that  could  arise  in  actual  implementation  of  such 
models,  oven  if  they  were  technically  precise  in  their  representation. 

The  documented  user  evaluation  and  subsequent  position  papers  (Appendix  F) 
speak  for  themselves  and  their  directness  would  perhaps  only  be  clouded 
by  further  elaboration  and  interpretation  here.  Nor  would  it  be  appro¬ 
priate  to  address  or  rebut  in  detail  exceptions  that  were  taken  with  the 
model  development  phase  (e.g.,  by  CNET  In  Appendix  F) .  This  is  more 
fittingly  a  task  falling  to  the  model  developers  and  project  directors. 

Rather,  the  comments  below  are  intended  to  put  the  various  user  comments 
into  perspective  so  that  a  clearer  image  of  user  feedback  is  provided 
for  the  purpose  of  test  and  evaluation. 

Taken  as  a  whole,  the  on-site  User  T&E  indicated  a  high  level  of 
consumer  receptivity  to  the  models  and  their  underlying  rationale. 

Here  the  models  were  viewed  primarily  in  terms  of  what  they  could  con¬ 
tribute  to  FTC,  Norfolk,  and  in  a  greater  sense,  to  COMTRALAN T .  Then 
the  models  were  also  viewed  in  terms  of  how  well  they  would  transfer 
to  similar  training  complexes  elsewhere  in  EDTRACOM.  The  possibility 
of  applying  the  models  within  CQMTRAFAC,  with  a  minimum  of  modification, 
was  generally  seen.  To  a  lesser  extent  the  models  were  viewed  as  a  system 
which  might  be  administered  from  the  CNET  throughout  EDTRACOM.  It  is 
doubtful  that  if  the  latter  viewpoint  predominated,  such  a  positive 
evaluation  would  have  been  rendered  by  the  users. 

CNTT's  final  position  (Appendix  F)  expressed  a  negative  view  of 
DOTS.  The  CNTT  comments  must  be  given  attention  in  view  of  the  fact  that 
the  technical  training  command  contains  the  largest  proportion  of  potential 
U6es  of  DOTS  type  products. 

The  strictly  technical  objections  raised  by  CNTT  are  addressed  in 
the  Management  Summary  in  a  later  section.  These  considerations  are 
technically  superficial  and  may  have,  in  turn,  stemmed  from  superficial 
explanation  of  these  features  at  the  on-site  T&E.  A  more  general  point 
was  made  that  the  FTC  development  site  was  unrepresentative  of  the 
critical  training  management  problems  and  was  a  poor  choice.  Consequently, 
proof  of  feasibility  was  not  seen  since  the  present  level  of  organizational 
application  is  not  cost  effective.  This  issue  is  exceedingly  relevant 
to  the  purpose  ot  the  T&E.  It  car.  be  dealt  with  only  throuRh  the 


realization  that  FTC,  Norfolk  was  not  t. hi-  perfect  sit'-  but  that  it  dees 
have  the  characteristics  that  were  necessary  and  sufficient  to  establish 
the  feasibility  of  the  modeling  approach  beyond  n  reasonable  doubt.  At 
a  minimum,  this  feasibility  was  establisnod  to  a  point  where  it  appears 
worthwhile  to  procted  to  a  rigorous  field  test  which  can  dispel  any  residual 
doubt . 


It  was  essential  that  the  models  produced  in  Phases  ]I  and  Hi  address 
the  schoolhouse  level  since  this  is  the  basic  building  block  of  the  EDTRACOM. 
Without  showing  a  basic  capability  to  deal  with  the  dynamics  of  this  level, 
the  demonstrated  feasibility  would  have  been  extremely  weak.  Yet,  as  CNTT 
has  pointed  out,  this  level  is  not  where  the  major  payoffs  are  to  be  gained. 
There  is  little  doubt  chat  it  would  have  been  much  better  to  have  also 
had  a  demonstration  of  modeling  applicability  at  a  higher  management  level 
(e.g.,  the  TRACOMS) .  A  field  test  of  the  applicability  cf  the  TPF  and 
SCRR  models  at  such  a  level  is  recommended  as  a  result  of  the  T&E. 

The  present  management  level  residence  of  the  model  is  just  one  of  many 
reasons  that  there  are  not  numerous  examples  of  actual  model  use  at  FTC, 
Norfolk.  Turnover,  training  time,  time  needed  to  gain  management  acceptance 
through  experience,  hesitancy  of  model  developers  to  trigger  defensive 
reactions  from  FTC  staff  (i.e.,  by  showing  virtue  of  potential  cuts),  and 
the  lack  of  a  well  defined  model  interface  position  (person)  all  contri¬ 
buted  to  the  low  usage  level.  It  is  significant  to  note  that  during  the 
follow-up  evaluation  at  COMTRAPAC  (see  preceding  section),  a  number  of 
real  world  problems  arose  toward  which  the  models  could  have  been  effec¬ 
tively  directed,  if  TRAPAC  data  had  been  compiled. 

The  structures  of  the  models,  including  even  the  ETE,  are  such  that 
there  are  no  major  conceptual  problems  apparent  in  adapting  them  to 
TRACOM  level  use.  It  is  expected  that  this  level  of  application  will  show 
high  utility  for  the  model  products.  Just  how  much  of  this  utility  can  be 
transferred  to  similar  functions  within  CNTT  cannot  be  determined  on  the 
basis  of  present  information.  It  is  clear  that  an  unwilling  user  would 
prohibit  any  such  transfer. 

Tt  cannot  be  helped  but  be  perceived  that  a  great  deal  of  the  negative 
CNTT  reaction  is  in  consideration  of  DOTS  as  a  "system"  and  not  as  a 
simple  of  tools  from  a  certain  branch  of  iiianagetiieiU  study.  The  "system" 
idea  is  not  a  mirage  since  there  are  presentations  relevant  to  this  point 
(e.g.,  TAEO  Report  No.  12-2,  Vul.  I,  Phase  II  Overview,  pp  1V-15  through 
I V— 19).  One  can  only  agree  with  CNTT  that  a  system  "for  all  levels  of 
management"  is  far  beyond  the  scope  of  what  has  been  demonstrated  thus  far 
(Appendix  F,  p.  F  —  1 7 ,  para.  3).  It  is  not  believed  that  CNTT  would  be  so 
quick  to  reject  the  models  as  tools  rather  than  the  primitive  form  of  a 
total  management  system. 


While  application  of  the  models  within  CNTT  proper  stems  unlikely  at 
present,  COMTRAPAC  emerges  as  a  desirable,  ready,  willing  and  able  field 
test  and  implementation  site  for  the  following  reasons:  (1)  favorable 
ill sposit ion  of  the  command  cowards  the  models,  (2)  a  strong  current  need 


for  improved  MIE  capabi  1  i  t  it--;  a”'’ 
making  tools,  (3)  present  access 


*1  °  y  OV.  ♦■»<■»  o  H  for  c  n  n  n  n  r  t  i\/r-  rlpr  i  nn- 

"  *"6  - - -  -  -  -  - 

to  all  necessary  terminal  equipment 


throughout  the  command,  (4)  general  familiarity  with  ADV  applications 
and  close  working  relationship  with  DPSCPAC. ,  (S)  existence  of  individuals/ 
positions  who  can  serve  as  effective  model  interface  points  for  mmuigers 
as  they  become  familiar  with  the  model  capabilities,  and  (6)  the  opportu¬ 
nity  presented  to  test  and  implement  the.  models  at  a  higher  management 
level. 

Financial  Feasibility  Assessment 

The  financial  feasibility  of  the  DOTS  models  was  addressed  by  the 
model  developers  (TAEG  Report  12-2,  Section  IV).  Obviously,  two  factors 
are  pertinent  here,  first  the  cost  of  formulating,  operating  and  maintaining 
and  second,  the  benefits  to  be  derived.  These  factors  must  be  considered 
in  the  light  of  the  scale  of  operations  (e.g.,  how  many  sites),  existing 
equipment,  existing  data,  opportunity /f requency  of  model  use,  and  future 
plans  for  the  EDTRACOM. 

Given  that  the  models  necessitated  the  acquisition  of  a  computer  main 
frame,  it  would  he  necessary  to  project  model  usage  beyond  one  site  in 
order  to  offset  the  initial  fixed  investment.*  The  bulk  of  the  mode) 
developer's  financial  analysis  is  thus  aimed  at  a  strategy  for  projected 
model  usage.  The  results  of  Phases  II  and  III  in  no  way  warrant  the  usage 
assumptions  made  in  that  analysis.  As  a  consequence,  the  resulting  coat- 
benefit  conclusions  are  not  viewed  as  acceptable. 


/.  V 

V  v. 


In  the  light  of  the  ubiquitous  time-sharing  resources  of  today 
and  the  competitive  pricing  of  those  services,  software  such  as  the  DOTS 
models  can  be  viewed  to  a  large  extent  as  independent  from  supDorting 
main  frames.  This  permits  the  cost-benefit  analysis  to  be  reduced  to  the 
more  concrete  parameters  of  the  model  development  site  and  its  immediate 
extensions.  Generalizations  can  then  be  made  to  other  potential  sites 
which  would  use  either  the  current  models  or  other  models  to  be  developed 
in  the  future.  (The  latter  case  would  also  Involve  development  cost.) 


As  a  result  of  discussions  with  members  of  the  User  T&E  team  and 
visits  to  various  EDTRACOM  sites,  it  can  be  validly  assumed  that  utili¬ 
zation  of  computer-based  models  like  those  of  the  DOTS  project  will  not 
require  special  data  or  equipment  acquisitions  of  a  significant  nature.** 
From  this  it  can  also  be  assumed  that  the  basic  skills  needed  to  operate 
the  models  will  also  be  present  <n  most  sites.  Since  model  operation  and 
data  maintenance  which  does  not  replace  existing  procedures  is  estimated  to 
consume  a  very  small  number  of  man-hours,  it  is  not  anticipated  that 
support  personnel  cost  will  be  appreciable.  There  will  he  a  training  cost. 
On  the  basis  of  training  conducted  prior  to  the  on-site  User  T&E,  this  is 


*That  is,  given  the  nature  of  models,  it  would  be  extremely  unusual  for 
even  a  group  of  them  to  justify  the  cost  of  a  dedicated  main  frame. 

**Th*>  facts  that  the  models  primarily  draw  on  data  presently  being  collected 
and  that  necessary  hardware  nas  eithei  beer,  acquired  or  soon  will  be, 
regardless  of  the  models,  have  been  discussed  earlier. 
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estimated  at  about  four  man-weeks  per  site  (2-4K).  Data  pertaining  to 
t  ime-shar lng  cost  for  the  models  were  obtained  from  actual  model  opcritl  ion 
at  FTC,  Norfolk  (see  TAEC  keport  10  May  197S,  Users  lest  Guide,  section  V). 
Using  these  cost  data  and  a  somewhat  arbitrary  estimate  of  the  extent  ot 
usage,  an  estimate  of  $10K  per  year  was  obtained  (using  all  three  models). 

With  some  amortization  of  training  cost,  this  results  in  a  total  yearly 
cost  that  can  be  safely  estimated  at  under  $15K,  allowing  for  some  eouip- 
ment  depreciation  (or  rental  cost  allocation)  and  model  maintenance. 

On  the  bene  £  it  side,  perhaps  the  most  meaningful  indicator  revealed 
in  the  T&E  was  provided  by  comments  from  the  User  T&E  team  that  the  modeling 
and  MIS  capabilities  demonstrated  by  DOTS  would  be  worth  at  least  two 
billets  at  the  training  center  level  of  implementation.  As  the  technical 
feasibility  section  of  this  report  pointed  out,  a  field  test  is  necessarv 
to  precisely  identify  a  measure  of  benefit.  However,  the  costs  as  estimated 
above  arc  low  enough  to  make  application  of  the  models  a  low  risk  proposition, 
especially  at  the  TRACOM  level. 

The  greater  question  of  financial  feasibility  concerns  to  what  extent 
the  results  of  DOTS  can  be  extrapolated  to  assess  the  feasibility  of 
applying  such  management  techniques  throughout  the  EDTRACOM  in  general. 

Since  the  techniques  employed  by  the  DOTS  models  (e  g. ,  LP,  simulation) 
are  a  fair  sample  of  the  field  from  which  they  are  drawn,  it  seems  reasonable 
to  generalize  the  order  of  operating  costs  estimated  above.  The  quality 
of  benefit  analysis  thus  far  possible,  in  no  way  warrants  similar  general¬ 
ization.  However,  a  plus  factor  for  products  such  as  the  DOTS  models  is 
the  current  state  of  management  information  systems  (MIS)  within  the 
EDTRACOM.  These  are  currently  emergent  and  embryonic.  Since  the  models 
are  very  complementary  to  the  MIS  development  process,  a  fairly  consistent 
MIS  benefit  might  be  projected  from  one  development  site  to  another. 

This  underlying  benefit  might  be  capable  of  covering  most  risks  associated 
with  the  more  sophisticated  utilities  of  models. 

Development  cost  must  also  be  computed  into  the  generalized  feasibility 
assessment.  The  costs  incurred  to  develop  the  DOTS  models  appear  inordinate 
and  probably  not  supportable  at  anything  less  than  command  level  implemen¬ 
tation.  But  those  costs  included  many  one-time  items  necessary  to  initia¬ 
tion  of  the  project.  It  is  believed  that  development  cost  could  be 
brought  to  a  reasonable  level  if  a  small,  in-house  modeling  capability 
was  pursued  by  ONET,  using  contractor  support  only  for  very  specific  and 
definable  end  products.  In  this  way  (1)  maximum  transfer  of  learning 
could  be  made  from  one  site  to  another,  (2)  the  sizeable  initial  cost  of 
becoming  acquainted  with  the  EDTRACOM  environment  would  not  be  repetitive, 
and  (3)  a  closer  and  more  continuous  user/model  developer  relationship 
could  evolve.  This  procedure  would  minimize  many  of  the  large  cost  items 
associated  with  model  development,  maximize  many  of  the  factors  which 
contribute  to  effective  modeling,  and  in  general  take  advantage  ot  the 
large  capital  base  presented  by  the  1  P'-'KACOM. 


Final  Statement  of  User  Evaluation 

At  the  conclusion  of  the  User  T&E ,  NAVPERSRANDCEN  prepared  a  final 
statement  of  user  evaluation.  This  management  summary,  which  includes 
recommendations  for  the  conclusion  of  DOTS  and  for  a  future  program 
resulting  from  DOTS,  is  presented  as  the  last  section  of  this  report. 


RESULTS  OF  TECHNICAL  REVIEW  OF  TECHNICAL  FEASIBILITY 


System  Capabilities/Requirements  and  Resources  Model  (SCRR) 


The  SCRR  is  a  linear  programming  (LP)  model  which  utilizes 
a  standard  software  package,  Mathematical  Programming  System 
Extended  (MPSX,  an  IBM  product).  It  deals  primarily  with  instructor 
and  classroom  resources  as  related  to  demand  (courses,  class 
input  capacity).  The  SCRR  was  developed  to  meet  four  specific  types 
of  management  problems.  These  are  discussed  below. 

1 .  Assessment  of  long-term  training  demand 

This  title  could  be  a  bit  misleading.  It  should  be  clear  that 
the  SCRR  is  not  a  predictive  or  probabilistic  type  model.  What  it 
can  do  is  take  near  or  long-term  training  demand  projections  as 
input,  and  then  operate  to  determine  whether  such  requirements  can 
be  feasibly  met  given  current  or  projected  resources  and  procedures. 

In  this  sense,  given  sufficient  lead  time,  it  can  aid  in  training  resource 
planning  and  capabilities  assessment. 

The  utility  of  the  model,  however,  does  not  depend  entirely 
upon  a  prerequisite,  accurate  demand  forecast.  The  model  could  be 
run  in  a  planning  mode  under  a  number  of  different  demand  assumptions. 
A  range  of  training  capabilities  and  resultant  impacts  could  thus  be 
assessed.  These  would  only  support  management  contingency  planning, 
however,  and  would  in  no  way  constitute  demand  forecasts  per  se. 

Another  product  of  the  SCRR  relevant  to  this  aspect  is  the 
"optimized  convening  rate.  "  This  is  the  number  of  course  convenings 
per  course  that  yields  maximum  student  throughput.  (This  is  a  result 
of  one  of  the  LP's  two  basic  modes  of  operation  -  the  other  being  the 
determin  ation  of  minimum  resources  given  fixed  convenings.)  This 
Optimal  Convening  Rate  would  seem  to  have  very  limited  utility.  It 
denotes  maximum  possible  output  only  under  assumption  that  all 
students  are  of  equal  value.  It  also  implies  a  free  market  type  environ¬ 
ment  and  one  in  which  demand  for  any  given  student  is  always  in  excess 
of  supply.  It  is  true  that  a  weighting  scheme,  if  extant,  could  be 
easily  input  to  the  model  thereby  yielding  a  meaningful  output  in  terms 
of  a  maximum.  But  the  determination  of  how  to  arrive  at  such  weightings 
with  appropriate  consideration  of  multi-level  training  system  objectives 
is  a  study  in  and  of  itself  and  one  for  which  the  basic  data  probably  are 
not  presently  available. 


2.  Assessment  nf  short-term  riemand  fluctuation  arising  from 


unscheduled  events  (e.g.  ,  a  Ship  repair  operation,  reserve  activation. 
unusual  seasonal  recruitment  levels). 


Here  different  resource  and  demand  configurations  can  be  "tried 
out"  to  assess  the  impact  of  such  perturbations  -  minimizing  their 
dysfunctional  effects  and  maximizing  the  responsiveness  of  training  - 
through  better  allocation  of  available  resources.  This  usage  seems 
quite  desirable  given  the  existence  of  frequent  perturbations  in  demand. 
Limited  usage  at  FTC,  Norfolk  seems  to  indicate  that  a  rather  minimal 
level  of  model  analysis  can  yield  information  of  broad  and  recurring 
utility  to  the  training  manager.  Yet  usage  of  the  models  by  managers 
seems  to  need  at  least  initial  stimulation. 


Assessment  of  training  resource  utilization 


Using  current  demand,  SCRR  can  compute  an  optimum  resource 
combination.  The  resulting  utilization  rates  may  be  contrasted  with 
current  utilization.  This  is  an  intuitively  attractive  feature  of  the 
model.  There  appear  to  be  potential  problems  in  interpretation  and 
ultimately,  actual  use  of  this  information.  The  utilization  rate  is 
presently  in  terms  of  hours  available  for  instruction.  The  instructor 
staff  at  the  schoolhouse  has  other  non-instructional  duties,  however. 
For  some  instructors,  these  duties  may  comprise  a  considerable 
portion  of  their  total  work  days. 

4.  Comparison  of  alternative  resource  allocation  strategies 


This  is  a  use  of  the  model  in  which  multiple  strategies  are 
evaluated,  given  known  performance  and  requirements  characteristics 
This  is  one  of  the  best  uses  of  the  linear  programming  capabilities  of 
the  models  and  could  precipitate  substantive  changes  in  present 
procedures  for  developing  plans  and  programs  as  well  as  justifying 
and  revising  the  same. 

Model  Selection 

The  decision  to  develop  an  LP  model  to  assess  capabilities  and  to 
allocate  resources  in  the  training  environment  was  sound.  The  relation 
ships  of  students  to  instructors  and  students  to  facilities  have  tradi¬ 
tionally  been  linear — although  in  a  stepwise  way.  Thus,  the  constant 
returns  to  scale  assumption  of  the  basic  LP  model  fits  well  into  the 
resource  management  framework  of  Navy  training. 

A  mixed  integer  formulation,  while  apparently  not  necessary  in  this 
application,  is  entirely  within  the  scope  of  this  type  of  model. 


The  information  -  rich  properties  of  the  LP  output  provide  a  fine 
opportunity  to  evaluate  potential  model  contributions  in  a  new  area 
of  application.  The  fundamental  LP  problem  characteristics  of 
proportionality,  additivity,  non-negativity,  divisibility,  and  deter¬ 
ministic  model  coefficients  are  adhered  to  by  this  application. 

Model  Formulation 


The  model  is  formulated  to  deal  with  variables  of  the  following 
types : 

course  variables:  length 

capacity 

convening  frequency 
instructor  requirements 
classroom  requirements 
lab  requirements 
equipment  requirements 

instructor  variables:  qualifications 

assignments 
availability 
rotation  date 

facilities  (class  and/or  Lab) 

variables:  location 

capacity 
availability 
course  assignment 

The  objective  function  is  to  maximize  student  throughput  (i.e.  , 
number  of  students  trained  per  model  run),  subject  to  a  variety  of 
requirements  constraints  related  to  the  variables  listed  above  (e.g.  , 
class  #  XXX  must  be  convened  Y  times  each  year).  A  continuous 
relationship  among  the  model  parameters  is  assumed  (i.e,  the 
problem  is  not  formulated  as  an  integer  program). 

The  model  is  well  formulated  from  the  standpoint  of  the  variables 
considered,  logical  representation  of  those  variables,  level  of  detail, 
and  type  of  information  produced.  The  overall  worth  of  the  model, 
however,  rests  very  heavily  upon  the  validity  of  the  "planning  factors" 
which  it  incorporates  (e.g.,  number  of  hours  instructors  are  available 
for  training,  student  -  instructor  ratio).  In  addition,  certain  needs 
to  segmentize  the  problem  within  the  LP  formulation  result  in  a 
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much  less  sensitive  model  than  might  be  expected.  The  resources 
and  requirements  of  one  school  are  treated  as  independent  from  those 
of  another  in  this  model  framework.  This  appears  to  be  somewhat 
less  realistic  a  portrayal  with  respect  to  facilities  than  to  instructors, 
(i.e.,  Lectures  do  not  normally  require  specialized  classrooms, 
regardless  of  subject  matter;  instructor  qualifications,  on  the  other 
hand,  can  lead  to  considerable  specialization  and  hence,  independence 
from  one  school  to  another.)  Finally,  another  problem  lies  in  the 
interpretation  of  the  primary  objective  function  (i.  e.  ,  max  student 
thruput)  and  of  resulting  information  products  (i.  e.  ,  instructor 
utilization).  Each  of  these  points  is  discussed  more  fully  below. 

The  issue  of  the  validity  of  using  the  planning  factors  incorporated 
into  the  SCRR  is  a  difficult  one.  In  the  classic  applications  of  LP, 
the  requirement  constraints  are  explicit  and  unquestioned  (e.  g.  ,  the 
basic  formulas  used  in  a  chemical  production  process).  *  This  validity 
carries  directly  over  to  the  results  of  the  LP  model  of  such  a  process. 
However,  when  the  process  being  modeled  is  very  much  dependent 
upon  human  resoarces,  precise  requirement  formulas  are  usually 
unavailable.  In  this  environment  the  question  of  requirements  is 
usually  a  very  prominent  one.  Any  misconceptions  that  a  "requirement'1 
model  introduced  to  this  environment  provides  better  basic  require¬ 
ments  information  than  previously  available,  must  be  carefully 
avoided.  **  Yet,  in  the  particular  case  of  the  SCRR,  there  is  little 
doubt  that  modeling  has  demonstrated  the  potential  for  increasing 
the  quality  of  management  information  at  the  FTC  as  well  as  for  the 
functional  command  level  above  it. 

The  manner  in  which  this  has  been  done  is  well  illustrated  by 
the  parametric  analysis  performed  on  the  impact  of  instructor 
availability  (TAEG  Report  12-2,  Vol  II,  pp  11-54  -  60).  There 
presently  is  no  firm  standard  for  instructor  available  hours  at  the 
FTC.  A  standard  of  10C0  hours  per  year  per  instructor  was  initially 
used.  This  figure  is  very  important,  obviously,  since  it  determines 
feasibility  and  utilization  of  resources.  While  the  model  can  offer 
no  improvement  in  the  identification  of  instructor  availability,  it  can 

-See  Charnes,  A.  and  Cooper,  W.  W.  Management  Models  and 
Industrial  Applications  of  Linear  Programming,  (Vols,  I  and  II), 

Wiley  1961 

■  *  It  is  not  even  correct  to  take  the  position  that  since  "soft"  planning 
factors  are  being  used  anyway,  a  model  that  incorporates  such 


can  via  na  wcircc  t h n  current  procedures  .  ?!?. nnin°  f -?  r I- n r ^ 
heuri  sti  call  y  utilized  by  managers  are  quite  different  in  effect  from 


those  which  lie  within  the  aura  of  a  computer  model. 
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be  used  to  show  what  would  be  the  effect  of  different  availabilities  on 
the  training  complex.  (It  was  shown  that,  ceteris  parabus,  availabilities 
of  700  hours  or  900  hours  could  not  meet  convening  requirements.  ) 
Similar  analyses  can  be  performed  to  illustrate  the  effects  of  changing 
student-instructor  ratios,  class  capacities,  instructor  qualifications, 
etc .  * 


The  segmentation  of  resources  and  requirements  within  the  SCRR 
limits  its  ability  to  optimize  across  the  training  center.  While  this 
convention  enables  most  candidate  FTC  problems  to  be  fit  within 
reasonable  computer  storage  limitations,  it  appears  to  be  a  less  than 
accurate  representation  of  what  is  possible  within  the  real  training 
environment.  It  may  be  that  some  of  the  greatest  management  payoffs 
within  the  FTC  are  to  be  gained  from  optimizing  resources  and  require¬ 
ments  across  schools.  This  type  of  analysis  could  be  approached  to 
some  degree  with  the  model  as  currently  formulated  (given  some 
recoding  and  restructuring  of  data).  Future  model  modifications 
could  permit  it  to  be  performed  on  a  broader  scale. 

"V. 

The  objective  function  of  maximizing  student  thruput  is  less 
intuitive  than  one  which  minimizes  resources  for  a  given  demand. 

(The  latter  is  possible  within  the  SCRR  but  is  given  less  emphasis  in 
the  documentation.  )  Both  types  suffer  from  the  inability  to  discriminate 
the  value  of  one  student  versus  another.  Thus  the  optimization  is  biased 
towards  shorter  courses  and  courses  which  are  resources  rich  (i.  e.  , 
a  student  is  a  student).  Utilization  rates  produced  by  the  SCRR  are 
difficult  for  users  to  interpret.  Aside  from  the  basic  consideration 
of  what  to  put  in  the  denominator  of  the  utilization  index  (i.  e.  ,  the 
availability  question  described  above),  one  is  hard  put  to  interpret 
the  figures  in  the  "good-bad"  framework  that  managers  seek.  For 
example,  an  increase  in  the  number  of  course  convenings  will 
increase  resource  utilization  but  could  decrease  course  utilization  (in 
actuality).  Also,  any  utilization  figure  must  be  interpreted  by  the 
user  with  due  consideration  for  available  time  consumed  by  supervisory 
duties,  military  duties,  preparation  for  instruction,  etc..  These  are 
factors  not  considered  by  the  SCRR  at  present. 


A  fault  of  the  Phase  UI  documentation  is  that  such  applications  are 
described  as  possible  yet  only  the  instructor  availability  analysis  is 
actually  documented  for  evaluation.  The  old  saw  of  a  picture  being 
worth  a  thousand  words  definitely  applies  here.  An  analysis  is  supposedly 
conducted  on  effects  of  instructor  cross  training  is  not  documented 
for  evaluation  at  all. 
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Model  Valldat ion 


The  basic  linear  procramminr  technoLocv  ami  the  MPSX 
software  are,  of  course,  not  in  need  of  validation  due  to  their 
standardized  nature.  The  validation  documented  by  the  model 
developer  (TAEG  Report  12-2,  Vol  II,  pp  11-30  -  60)  demonstrate 
that  the  model  is  functioning  properly.  The  model's  ability  to 
accurately  represent  the  real  world  is  not  demonstrated.  This  must 
be  done  wi*h  historical  data  or  under  field  test  conditions.  In  the 
absence  of  such  information,  the  SCRR  is  judged  to  be  a  useful  and 
practical  representation  of  the  training  complex.  * 


■■This  judgement  is  a  function  of  (1)  feedback  from  prospective  users 
obtained  in  the  user  T&E  and  other  discussions  (2)  a  comparison  of  the 
model  s  formulation  with  a  description  of  the  training  complex  operation 


Training  1‘rocrss  Flow  Model 


The  objective  of  the  TPF  is  to  provide  a  means  for  simulating 
the  fl  ow  of  students  through  a  training  complex.  This  simulation  is 
intended  to  enable  managers  to  assess  the  effects  of  changes  in  the 
primary  variables  of  the  training  complex  and  its  immediate  environ¬ 
ment.  The  model  operates  on  two  categories  of  data,  those  pertaining 
to  students  (10)  and  those  pertaining  to  courses  (15).  Given  inputs 
such  as  maximum  or  average  course  capacity,  number  of  convenings, 
and  demand,  the  TPF  computes  course  utilization,  backlog,  AOB, 
etc.  ,  showing  these  on  a  quarterly  bases  through  time. 

The  variables  currently  dealt  with  by  the  TPF  have  been  screened 
from  a  larger  set  of  variables  on  the  basis  of  statistical  analyses. 

An  original  objective  of  the  TPF  was  to  develop  statistical  parameters 
from  these  analyses  which  could  be  included  in  the  model  for  the 
purpose  of  predicting  training  results  as  a  function  of  student  profile 
data.  Correlations  resulting  from  these  statistical  analyses  were  not 
of  sufficient  significance  to  warrant  inclusion  in  the  model  and  there¬ 
fore  were  not  incorporated.  Straightforward  proportionality  figures 
are  currently  used  by  the  TPF  to  assess  the  effects  upon  the  training 
system  of  changes  in  demand,  scheduling,  or  capacities.  Failure 
rate,  no-shows,  disc nrollments ,  etc.,  are  not  presently  forecasted 
by  the  model  on  the  basis  of  student  profile  data.  Thus  the  TPF,  as 
a  simulation,  mathematically  represents  the  mechanics  of  the  training 
center  scheduling  function  from  a  deterministic  rather  than  probabilistic 
basis. 

Model  Selection 

The  choice  of  a  process  flow  model  was  a  good,  if  perhaps  obvious, 
selection  (and  as  such  did  not  depend  greatly  on  the  results  of  Phase 
I) .  The  course  scheduling  problem  is  one  which  both  lends  Itself  to 
modeling  and  at  the  sane  time  represents  a  significant  training  manage¬ 
ment  problem.  The  flow  also  offers  a  complementary  capability  to  the 
resource  allocation  model  (SCRR)  that  was  concurrently  developed. 

The  magnitude  of  original  potential  payoff  from  this  model  is 
substantially  reduced  as  a  result  of  its  present  inability  to  statistically 
relate  student  profile  data  to  student  behavior  within  the  training 
system  and  the  consequent  inability  to  gauge  the  potential  impact  of 
such  factors  (see  below). 
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Another  restrictive  factor  is  the  apparently  limited  opportunity 
which  training  managers  at  the  FTC  level  have  to  utilize  such  a  model 
in  a  proactive  manner.  This  mostly  stems  from  the  fact  that  fleet 
training  demands  are  at  the  same  time  hard  to  predict  and  hard  to 
deny.  On  the  other  hand,  the  more  reactive  capabilities  of  the  model 
(e.  g.  ,  to  answer  ''what  if  questions)  do  not  appear  to  be  in  high 
demand  at  the  FTC  level,  though  they  may  be  periodically  of  substantial 
value  to  that  level.  It  is  important  in  considering  this  to  keep  in  mind 
the  DOTS  models  were  selected  on  the  basis  demonstrating  the  feasibility 
of  an  approach  for  CNET  and  not  on  the  basis  of  what  models / techniques 
would  yield  the  highest  payoff  to  FTC,  Norfolk  in  particular. 

In  contrast  to  the  time/flow  orientation  of  the  TPF,  the  SCRR 
offers  more  of  a  snapshot  assessment  of  the  capabilities  and  require¬ 
ments  of  the  training  system  at  a  given  point  in  time.  It  adds  the 
considerations  of  facility  and  instructor  resources  to  the  problem  of 
"How  many  classes  of  what  size  can  be  convened9  These  considera¬ 
tions  form  a  necessary  complement  to  the  TPF.  In  fact  answers 
from  either  model  would  be  partially  naive  without  those  of  the  other. 

This  prompts  the  question  of  why  select  and  separately  develop  two 
models,  neither  of  which  is  sufficient  by  itself,  rather  than  develop 
one  integrated  model  from  the  beginning.  The  answer  to  this  question, 

I  believe,  lies  in  the  initial  orientation  of  the  project  -  it  has  been 
technology  oriented  rather  than  problem  oriented  (i.  e.  ,  "test  the 
feasibility  of  applying  new  decision  making  technologies").  Developing 
two  separate  models  provided  a  more  secure  approach  to  that  problem. 
Furthermore  the  original  concept  of  the  TPF  (i.  e.  ,  to  forecast 
training  complex  inpact  as  a  function  of  student  profile  data)  lent  it 
more  viability  as  a  separate  model  concept. 

Given  the  present  lack  of  statistical  relationships  with  student 
data,  present  functions  of  the  TPF  may  have  been  more  easily  carried 
out  by  selecting  some  ether  type  of  approach  such  as  using  a  data 
base  management  package.  Nevertheless,  much  of  the  same  analysis 
that  was  required  to  formulate  the  TPF  would  similarly  have  been 
requi red . 

Model  Formulation 

The  TPF  is  formulated  so  as  to  simulate  on  a  week  by  week  basis. 
Provision  is  made  for  holidays.  Since  a  course  can  only  be  directly 
shown  as  convening  a  maximum  of  50  times  (one  per  week)  in  this 
manner,  duplicate  courses  are  created  within  the  scheduling  algorithm 
in  order  to  handle  convenings  in  excess  of  that  number. 


Rased  on  the  technical  information  presented,  it  appears  that 
an  alternative  formulation  might  be  more  direct.  The  need  for  "week 
by  week"  information  was  not  apparent  from  either  the  DOTS  documen¬ 
tation  or  feedback  from  users  at  FTC.  Given  this,  the  basic  scheduling 
dynamics  of  the  FTC  that  were  derived  from  the  training  process  analysis, 
could  have  been  represented  in  the  model  as  a  vector  that  could  be 
transformed  by  various  conditional  data  to  yield  output  on  a  quarterly 
basis,  (i.  e.  ,  There  is  no  competing  reason  to  maintain  the  "week  by 
week"  level  of  detail  -  especially  if  it  demands  more  complex  programming 
and  data  manipulation.  ) 

Though  the  TPF  does  not  currently  simulate  on  the  basis  of 
student  profile  data  and  the  results  of  the  statistical  analysis  of  that 
kind  of  data  are  not  incorporated  into  the  model,  some  comments 
are  warranted.  First,  the  idea  of  formulating  a  model  to  predict 
impact  on  the  training  process  from  various  types  of  student  inputs 
remains  intuitively  desirable.  Users  participating  in  the  on-site  T&E 
indicated  that  large  payoffs  could  result  from  such  an  ability.  Further¬ 
more,  while  the  DOTS  statistical  analysis  failed  to  uncover  correlations 
of  sufficient  significance  to  warrant  inclusion  in  the  TPF,  the  conduct 
of  that  analysis  does  not  support  a  conclusion  that  such  correlations 
are  not  to  be  ohtained. 

A  statistical  analysis  such  as  that  used  in  the  course  of  developing 
the  TPF,  should  be  preceded  by  the  development  of  a  conceptual  model 
(reduced  to  mathematical  form)  which  states  the  relationships  that 
are  hypothesized  to  exist.  The  danger  of  proceeding  without  such  a 
model  are  many.  Examples  can  be  seen  in  the  tables  of  results 
documented  from  the  DOTS  analyses  (e.  g.  ,  TAEG  Report  12-2 ,  Vol  III, 
p.  63).  Another  manifestation  is  in  the  representation  of  failure  in 
the  regression  model.  The  statistical  analysis  used  data  which 
described  failure  as  a  0,  1  variable.  This  in  itself  is  a  violation  of 
assumptions  underlying  the  use  of  regression  in  the  first  place.-'  A 
second  point  is  that  the  model  is  constructed  from  an  individual 
standpoint;  the  dependent  variable  is  "did  the  student  pass  or  fail," 
the  independent  variables  being  that  student's  GOT  and  ARI  scores  and 
the  number  of  schools  previously  attended.  Yet  the  resulting  model 
is  used  to  predict  a  grovip  failure  r..te  on  the  basis  of  average  GCT, 


•See,  for  example,  Ncter,  John  and  Mayncs,  E.  S.  ,  "On  the 
appropriateness  of  the  correlation  coefficient  with  0,  1  dependent 
variable.  "  Journal  of  the  American  Statistical  Association, 

June  1970,  Vol  06,  No  33U,  pp  501  -509. 


ARI,  and  number  of  schools  for  the  group.  This  is  tin  inappropriate 
use  of  the  coefficients  derived  from  the  regression  analysis  performed 
to  construct  the  model.  Finally,  it  deserves  to  be  pointed  out  that 
if  the  objective  was  to  predict  group  failure  rates,  a  number  ol  other 
variables  should  have  been  included  in  the  analysis  (e.  g.  ,  instructor 
variables,  group  mis,  class  size,  etc.  ).  :;- 

Model  Validation 

The  validation  procedures  for  the  TPF.  as 
described  by  the  model  developer  (TAEG  Report  12-2,  Vol  II,  IV, 
pp,  22-30)  do  not  constitute  a  true  validation.  The  model  is  simply 
exercised  in  a  number  of  conditions  and  its  deterministic  processes 
are  allowed  to  respond  accordingly.  It  is  clearly  demonstrated  that 
the  model  is  in  operating  order.  However,  there  is  no  basis  for 
judging  how  well  the  model  simulates  the  real  training  process  flow. 

All  that  can  be  said  is  that  the  factors  utilized  in  the  TPF  appear 
to  be  reasonable  since  they  arc  drawn  from  those  which  are  commonly 
used  by  the  training  managers. 

Of  course,  a  true  validation  would  have  entailed  the  use  of 
historical  conditions  and  the  matching  of  model  output  to  actual  out¬ 
comes  in  the  training  environment  resulting  from  those  conditions. 
There  appears  to  have  been  no  attempt  at  sm  h  a  procedure.  (Of 
course,  this  would  be  more  of  a  validation  of  the  planning  factors 
used  by  the  model  than  of  the  mechanics  of  the  model  per  se.  )  Thus 
we  know  that  the  TPF  is  a  useful  automated  version  of  the  kind  of 
scheduling  that  is  presently  being  carried  out  at  the  FTC.  We  have 
a  more  flexible  and  quick  means  for  developing  schedules  and  assessing 
demand  impacts.  It  has  not  been  demonstrated  that  we  have  a  more 
precise  or  accurate  means  for  making  such  developments  or 
asses  sine  nts . 


For  more  elaboration  of  these  points,  see  Appendix  G 
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Educational  Technology  Evaluation  Model  (ETE) 


The  ETE  is  an  entity  flow  simulation  model  for  ILS  course  design 
and  management.  Like  the  TPF  and  SCRR,  this  model  is  intended  to 
answer  questions  concerning  capabilities  (capacity),  utilization,  and 
resource  requirements  as  well  as  course  completion  time.  It  has  the 
capability  to  evaluate,  in  compressed  time,  the  effects  of  real  or 
projected  changes  in  resources  or  demand.  Since  equipment  and 
personnel  can  be  costed  external  to  the  model,  the  model  can  be  used 
in  making  cost-effectiveness  determinations  of  different  ILS  course 
designs. 

The  ETE  is  a  general  purpose  simulation  model  and  is  thus  not 
structured  around  a  specific  type  of  course.  Its  data  base  would  not 
normally  even  approach  the  complexity  of  that  of  the  TPF  and  SCRR. 
l'he  data  for  a  typical  model  run  can  be  entered  by  the  user  at  the 
.^rt  of  a  model  session  (and  then  stored  for  later  recall  or  modifica- 
„:.on).  Theoretically,  the  ETE  could  at  present  be  applied  to  any  ILS 
course  or  group  of  courses,  regardless  of  location  or  even  if  the 
course  is  yet  to  be  convened. 

Of  course,  the  ETE  must  work  from  basic  requirements  informa¬ 
tion  just  like  most  models.  For  example,  the  viser  has  to  input  the 
number  of  hours  of  instructor  time  needed  for  x  number  of  students, 
the  average  time  required  by  a  student  on  a  certain  module,  the 
arrival  rate  of  students  (i.  e.  ,  demand)  etc. 

Model  Selection 

Factors  favoring  the  development  of  this  model  included:  (1) 
increasing  numbers  of  ILS  courses  throughout  EDTRACOM,  (2)  increasing 
resource  constraints,  (3)  existence  of  previous  work  in  simulating  ILS 
in  the  Navy,  (4)  prospects  for  a  general  purpose  model  that  could  be 
put  to  use  immediately  throughout  EDTRACOM,  and  (5)  emphasis  of  other 
models  on  training  process,  as  opposed  to  course  design. 

Model  Formulation 

The  RTF.  makes  use  of  a  standard  IBM 
programming  product  called  General  Purpose  Simulation  System 
(GPSSV).  This  is  a  widely  recognized  and  utilized  simulation  scheme. 
The  model  presents  a  good  adaption  of  the  ILS  environment  to  the 
simulation  framework.  All  of  the  major  ILS  decision  variables  can 
be  represented  and  considerable  user  flexibility  is  provided.  The  ETE 
is  generally  formulated  so  that  the  user  can  formulate  a  specific 
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model  to  represent  a  given  ILS.  The  estimated  time  for  such 
formulation  (3-5  days)  is  acceptable.  (See  TAEG  Report  12-2, 

Vol  II,  pp  III-26,  31.  ) 

Model  Validation 

In  the  absence  of  II, S  at  FTC  Norfolk,  the.  EXE 
was  validated  on  the  basis  of  a  proposed  consolidated  EW  school  at 
Corry  Station.  The  validation  was  not  in  terms  of  how  well  the  ETE 
simulation  represented  actual  course  behavior,  but  rather  to  the 
extent  that  results  replicated  those  obtained  from  the  application  of 
a  previously  developed  special  purpose  simulation  model  of  the  EW 
s  chool. 

This  procedure  offers  certain  advantages  (e.g.  ,  the  opportunity 
to  obtain  indicators  of  both  model  generality  and  sensitivity).  It  nas 
other  rather  strong  disadvantages  (e.  g.  ,  insulation  from  a  "real  world 
comparison,  anchoring  to  another  model  of  undefined  validity  and 
fidelity,  absence  of  user  based  utility  information).  Aside  from  the 
variability  analysis  which  is  acceptable  (TAEG  Report  12-2,  Vol  II, 
p  III-24),  it  is  only  known  that  the  ETE  is  as  good  as  or  better  than 
another  simulation  model  as  applied  to  the  EW  school  use.  Nor  does 
this  validation  procedure  establish  the  generality  of  the  ETE  as 
implied  by  the  model  developer  (TAEG  Report  12-2,  Vol  II,  pp  23-24). 

As  a  result,  the  ETE  remains  for  the  most  part,  in  an  unvalidated 
state.  It  may  be  that,  as  the  model  developer  states,  there  are  no 
NAVEDTRACOM  ILS  applications  that  are  presently  suitable  for  use 
in  validation.  If  this  is  still  true,  the  condition  should  not  prevail  for 
long.  The  ETE  should  be  validated  against  a  number  of  different 
actual  IES  cases  as  soon  as  possible. 

With  the  exception  of  the  model  validation  procedure  (which 
may  have  been  a  necessity),  the  ETE  appears  to  have  been  soundly 
developed  from  a  technical  viewpoint.  However,  this  technical 
soundness  is  as  of  yet,  not  bulwarked  by  a  clearly  demonstrated 
utility  (i.  e.  ,  a  utility  determination  arrived  at  through  interaction 
with  actual  ILS  designers  and  managers). 

Since  the  basis  of  comparison  is  a  "special  purpose,  ad  h  >c  model" 
in  the  fi  rst  place. 


General  Comments  on  Validation  Procedures  Utilized  in 
Phase  III  and  the  T&E 

In  general,  the  quality  of  the  validation  procedures 
for  the  DOTS  models  conducted  during  both  model  develop¬ 
ment  and  the  T&E,  suffers  from  the  lack  of  a  legitimate 
field  test.  Consequently,  an  issue  for  the  T&E  is 
whether  or  not  a  field  test  should  have  been  conducted 
in  Phase  III.  It  is  the  judgement  of  this  T&E  that  such 
a  field  test  could  not  have  been  comprehensively  per¬ 
formed  in  Phase  ill  for  the  following  reasons:  (1)  The 
models'  internal  logic  and  programming  were  still  being 
tested,  (2)  Users  had  not  had  time  to  become  intimately 
familiar  with  the  models'  capabilities,  (3)  Data  manage¬ 
ment  capabilities  and  supportive  terminal  equipment  at 
the  FTC,  as  well  as  elsewhere  in  the  EDTRACOM  were  at  an 
embryonic  stage  (these  capabilities  are  prerequisite  to 
effective  model  usage),  and  (4)  The  models  had  yet  to  be 
evaluated  by  potential  users  for  general  reasonableness 
and  acceptability.  (It  would  not  have  been  possible  to 
obtain  the  user  commitment  necessary  for  proper  field 
testing  without  first  being  able  to  demonstrate  that  the 
models  were  in  working  order,  that  they  could  be  made  to 
operate  in  the  training  environment,  and  that  they 
addressed  substantive  training  issues  in  a  useful  and 
meaningful  manner.)* 

Present  conditions  for  field  resting  of  the  SCRR  and 
TPF  appear  to  be  very  good  at  both  the  COMTRALANT  and 
COMTRAPAC  sites.  Furthermore,  prospects  for  usable  end 
products  emerging  from  such  testing  are  sizable.  This 
field  testing  should,  at  a  minimum,  fill  in  some  of  the 
blanks  left  from  this  T&E  evaluation  regarding  more  pre¬ 
cise  estimates  of  cost,  accuracy,  and  utility.  A  suitable 
field  test  site  for  the  ETE  model  is  not  so  evident,  but 
must  be  pursued.  It  would  not  be  wise  to  put  that  model 
on  the  shelf  and  wait  for  1980  to  happen. 


*These  conditions  did  not  rule  out  validity  resting  using 
histonca)  data.  Unfortunately,  the  historical  data 
available  would  not  have  been  complete  enough  to  allow 
even  this  type  of  testing. 


itcttr  tv  am  EVALUATION  OF  OPERATIONAL  FEASIBILITY 


Potential  Model  r.nnt-rl Kuf>f>r.«i  (First  Msmssion  Area)* 


General  areas  of  model  contribution  were  noted  as 
follows : 

1.  Assessment  of  current  training  organizations 

2.  Monitoring  ongoing  training  activities 

3.  Evaluating  possible  effects  of  changes  in 
existing  training  systems 

In  addition  to  more  effective  performance  of  the  training 
management  function  as  currently  practiced,  the  models 
have  the  potential  for  introducing  new  management  tech¬ 
niques  and  supporting  certain  types  of  decision  making 
not  currently  practiced.  The  DOTS  developments  are  seen 
as  contributing  in  two  distinct  but  interdependent  modes. 
First,  as  a  management  information  system  by  virtue  of 
the  data  base  management  capabilities  developed  to 
support  the  models.  This  was  not  a  direct  objective  of 
DOTS  but  was  a  prerequisite  to  producing  models  which 
can  effectively  function  in  the  operational  environ¬ 
ment.  Second,  as  a  management  decision  support  system 
to  be  used  in  the  kinds  of  functions  noted  above.  More 
specific  evaluation  of  potential  model  contributions  is 
presented  further  below. 

The  following  command  level  problem  areas  were 
identified  by  FTC,  Norfolk  personnel  in  December  1974 
as  arc?as  in  need  of  additional  support  (not  necessarily 
areas  seen  as  amenable  to  model  development) .  These 
were  evaluated  in  the  user  T&E  using  a  specific  set  of 
criteria  related  to  potential  model  contribution  (see 
Appendix  E) .  Summary  results  are  indicated. 


*Much  of  the  information  for  this  section  was  gathered 
from  the  application  of  a  consistent  set  of  criteria 
to  a  number  of  test  problem  areas.  The  criteria  were 
applied  by  each  of  the  User  T&E  team  members  and  the 
Acting  Director  of  Training  (DOT)  at  FTC,  Norfolk, 
(bee  Appendix  "E"). 


1.  Instructor  Personnel  Recap  Reports  (e.  g.,  instructor 
allowance  list,  assigned  vs.  allowance,  instructor 
requirements  and  assignments  by  course,  etc.). 

This  information  problem  area  was  seen  as  occurring 
on  a  monthly  basis  and  to  be  of  substantial  magnitude. 
Present  methods  were  seen  as  being  adequate  most  of  the 
time  for  addressing  this  area.  It  was  indicated  that 
the  problem  area  is  likely  to  persist  into  the  future. 
Views  on  the  opportunity  for  improvement  were  mixed, 
ranging  from  small  to  substantial.  Possible  dollar 
savings  on  this  area  were  seen  as  generally  minimal. 
Non-monetary  benefits  were  seen  as  moderate.  The  need 
for  changes  to  current  organization  in  order  to  implement 
improvements  was  seen  as  minimal.  The  model  data  base 
seen  as  quite  applicable  to  this  area. 

2 •  Determining  Impact  of  Changes  in  Student  Throughput 
Upon  Instructor  Requirements  (within  available  activity 
resources) . 

These  kinds  of  problems  seem  to  occur  semi-annually 
or  at  most,  monthly.  Their  magnitude  ranges  widely. 

They  are  likely  to  be  extant  well  into  the  future.  The 
opportunity  for  improving  problem  solving  capabilities 
here  is  seen  as  substantial  with  prospective  savings 
estimated  in  the  10K  to  100K  range  annually  per  activity. 
Present  methods  were  seen  generally  as  being  inadequate 
most  of  the  time  for  these  types  of  problems.  The  SCRR 
and  TPF  models  were  seen  as  applicable  here. 

3 .  Determining  Effect  of  Changes  in  Contact  Hours  on 
Instructor  Requirements  and  on  the  FTC  in  General, 

This  problem  type  occurs  yearly  or  semi-annually 
and  is  of  substantial  magnitude.  Present  methods  are 
usually  inadequate  but  opportunity  for  improvement  is 
seen  as  quite  variable,  depending  on  the  command  in 
question.  (Dollar  savings  projected  at  10-100K  per 
command  per  year.)  Moderate  organizational  changes  were 
seen  as  being  required  to  implement  improvements. 

The  data  base  and  SCRR  model  were  seen  as  applicable 
to  these  problems. 
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Predictin 


Course  Length  Data. 


There  was  highly  mixed  assessment  of  this  problem 
area.  Data  obtained  at  FTC,  Norfolk  failed  to  produce 
statistical  correlations  for  formal  school  training  of 
sufficient  significance.  It  was  thought  that  research 
into  this  area  should  continue.  Presently,  the  DOTS 
models  do  not  adequately  address  this  area,  though  they 
might  be  made  to  do  so. 

5.  Assessinq  Present  or  Potential  Effects  of  instructor 


Cross  Trainin 


Problem  could  occur  semi-annually  and  is  of  sub¬ 
stantial  magnitude.  Present  methods  are  usually 
inadequate  and  opportunity  for  substantial  improvement 
is  present  ($  savings  in  100K  range  yearly  per  activity) . 
Small  to  moderate  changes  in  current  organization  and 
policy  are  seen  as  necessary  to  implement  improvement. 
Data  base  and  SCRR  model  are  seen  as  addressing  this  area 

6.  Effects  of  Trainincr  Staff  Personnel  cuts  upon  orqani- 


zation  Capabilities. 

This  problem  occurs  monthly  to  semi-annually  and  is 
of  critical  magnitude.  Present  methods  are  generally 
seen  as  adequate  most  of  the  time  with  mixed  views 
existing  as  to  opportunity  for  improvement.  Possible 
dollar  savings  rang -s  from  zero  to  millions.  This  problem 
area's  likely  to  persist  into  the  future.  Moderate  to 
substantial  organizational  changes  are  seen  as  necessary 
to  implement  improvements.  Data  base,  SCRPv  and  TPF 
models  are  viewed  as  applicable. 

7.  Effects  of  Chanqes  in  Convening  Frequency  Upon 


Present  methods  seen  as  inadequate  with  substantial 
opportunity  for  improvements  (possible  savings  in  100K 
to  million  dollar  range) .  Minimal  changes  in  present 
organizational  policies  art,  required  to  implement 
improvements.  Frequency  of  problem  occurrence  seen  as 
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and  Staff  Requirements 


This  problem  type  was  seen  as  seldom  occurring  at 
FTC,  Norfolk.  Other  reactions  (apparently  based  on 
other  training  sites)  indicated  monthly  to  yearly 
occurrence  of  major  magnitude.  Present  methods  are 
viewed  as  generally  inadequate  with  opportunity  for 
improvement  depending  on  command  (10K  to  million  range) . 
Moderate  changes  in  present  policies  are  required  to 
take  advantage  of  opportunity.  SCRR  model  is  seen  as 
applicable  here. 

9.  *Assessing  Resource  Interrelationships  (e.  g.,  effect 
of  changes  in  one  resource  upon  another) . 

This  problem  is  of  major  magnitude  with  current 
methods  usually  inadequate  at  TRACOM  level  but  usually 
adequate  at  CNTT.  This  type  of  problem  will  persist 
into  the  future,  presenting  great  opportunity  for 
improvements  ( 100K  to  million  dollar  range).  Moderate 
organizational  changes  will  be  required  to  implement 
improvements.  Problem  is  of  daily  nature. 

SCRR  ar  '  TPF  models  are  seen  as  applicable.  However, 
these  models  are  not  currently  configured  to  conveniently 
answer  certain  types  of  these  questions  such  as  inter¬ 
active  effects  of  convening  frequency  vs.  class  size,  or 
the  assessment  of  interactive  effects  over  a  range  of 
variable  values.  Programming  additions  would  be  required 
to  more  effectively  meet  this  information  need. 


♦These  last  three  problem  areas  were  not  among  those 
initially  identified  by  FTC,  Norfolk  managers.  They 
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contribution . 


10 .  *1 LS  Design 

This  problem  area  currently  exists  to  a  major  degree 
in  CNTT  and  will  do  so  in  the  future  at  the  TRACOMS . 
Present  methods  are  seen  as  inadequate  most  of  the  time 
with  substantial  to  great  opportunity  for  improvement 
(million  dollar  area) .  Moderate  to  substantial  changes 
in  present  organization/policies  are  seen  as  necessary 
to  effect  improvements. 

11.  Other  Factors  Affecting  Contribution 


A  major  consideration  in  the  realization  of  model 
potential  was  determined  to  be  the  level  and  manner  of 
model  use.  While  operation  of  the  model  mechanisms  per 
se  (e.  g.,  terminal  operation)  is  not  complex,  use  of 
model  results  requires  all  of  the  skills  of  the  experienced 
training  manager.  Typical  model  support  requirements 
(e.  g..  documentation,  ease  of  use)  were  viewed  as 
essential  to  potential  realization.  (See  "Ease  of  Use" 
and  "Organizational  implementation"  sections  that  follow.) 

Several  participants  viewed  the  bulk  of  model  potential 
to  lie  in  application  at  the  "functional"  level  and  above 
(i.  e.,  "models  and  necessary  support  may  not  be  worth 
it  at  schoolhouse  level") .  On  the  other  hand,  partici¬ 
pants  with  "schoolhouse"  level  experience  expressed  the 
view  that  considerable  opportunity  exists  to  utilize 
model  potential,  given  "a  little  managerial  imagination". 

A  concise  assessment  of  actual  contribution  can  only  be 
obtained  via  a  field  test  (which  FTC,  Norfolk  was  not) . 

A  possible  hindrance  to  vertical  expansion  of  model 
application  is  the  plethora  of  overlapping  management 
systems  which  reside  at  the  upper  management  levels. 

While  the  integration  of  these  systems  is  not  within  the 
scope  of  DOTS,  higher  level  implementation  of  DOTS 
development  may  be  jeopardized  nevertheless.  Coping  with 
this  dilemma  remains  a  future  challenge  to  the  DOTS 
|  project  management. 
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A  recurrent  theme  of  the  T&E  discussions  centered 
on  the  NITRAS  data  system  (i.  e.,  its  shortcomings). 
The  complementary  character istics  of  the  DOTS  data 
base  management  system  was  recognized.  Specifically, 
DOTS  presents  a  way  for  effectively  utilizing  the  vast 
NITRAS  data  base  in  a  proactive  way. 


Ease  and  Practicality  of  Use  (Second  Ptscus9lon  Area 


User  Knowledge  Requirement 

Training  in  use  of  the  DOTS  model  will  be  needed 
at  a)l  training  command  levels,  though  to  varying 
degrees.  Essentially  three  levels  of  training  are 
required.  These  are  familiarization  for  upper  manage¬ 
ment,  system  analysis  for  translating  upper  management 
or  command  training  problems  into  model  parameters,  and 
a  systems  operation  for  training  those  who  would  operate 
hardware  components  of  DOTS.  The  familiarization  course 
would  emphasize  applications  of  DOTS  in  problem  solving, 
decision  making,  and  report  generation.  The  system 
analysis  course  would  provide  an  in-depth  applications 
course  emphasizing  the  flexibility  of  DOTS  in  problem¬ 
solving,  program  modification,  and  data  base  maintenance. 
The  systems  operation  course  would  be  a  fall-out  from 
the  system  analysis  course  (without  the  systems  analyst 
function) .  The  system  operation  course  would  be  centered 
around  hardware  operation  —  learning  to  operate  input 
and  output  devices  and  update  the  data  base. 

The  requirement  that  the  systems  analyst  should  be 
able  to  translate  problem  areas  and  give  preliminary 
interpretation  to  DOTS  output,  dictates  that  extensive 
knowledge  of  training  command  policies/functions  will  be 
needed  by  the  analyst.  DOTS  model  usage  skills  should 
be,  of  course,  secondary  to  training  command  expertise. 
Programming  experience  is  needed  in  order  to  modify 
models  or  for  inputting  special  data  onto  temporary 
(scratch)  files.  Systems  operator  would  r.ot  have  to  have 
special  skills,  although  familiarity  with  computing 
system  operations  would  probably  be  useful. 

In  most  cases  (except  at  central  configuration 
management  level),  system  analysis  and  system  operation 
would  be  collateral  duty  for  the  personnel  involved. 
Identification  of  personnel  for  analyst/operator  functions 
and  training  of  these  personnel,  would  be  a  command 


responsibility.  The  systems  operator  course  would 
probably  be  about  2  weeks  long,  while  the  system 
analysis  course  would  last  2-4  weeks  (especially  if 
programming  and  data  modif ication  skills  are  to  be 
taught) .  The  familiarization  course  would  last  1-2 
days.  A  recommendation  was  made  to  incorporate 
familiarization/system  analyst  courses  into  training 
manager  schools  (as  part  of  "hew-to"  models  training) . 

Training  for  the  system  operator  would  be  made  easier 
if  editing  and  interactive  programming  were  standardized 
among  the  three  models.  Also,  a  more  extensive  pro¬ 
cedures  manual  ("cookbook")  is  needed,  for  system 
operators/analyst.  The  manual  should  describe  more 
applications  (problem)  areas,  and  procedures  required  to 
operate  the  DOTS  system  in  association  with  these 
specific  problem  areas.  A  more  accessible  data  elements 
dictionary  is  required  for  more  efficient  data/program 
updates,  vei if ication  and  modification  by  the  systems 
analyst.  In  addition,  a  basic  guide  which  details 
assumptions  and  fixed  parameters  of  each  model  should 
be  provided,  especially  for  more  accurate  interpretation 
of  model  output  by  upper  management  and  systems  analysts. 
These  manuals  and  guides  would  also  be  useful  in  training. 


Standardization  of  editing  and  interactive  program¬ 
ming  among  the  three  models  is  especially  necessary  to 
recover  efficiently  from  default  situations.  As 
currently  configured,  it  is  too  difficult  to  recover  if 
errors  occur  or  modifications  are  required  because 
edit/re-edit  procedures  must  be  repeated,  and  these  pro¬ 
cedures  are  highly  specialized  for  ea-h  model.  Some 
capability  to  restart  without  resetting  previous  condi¬ 
tions  should  also  be  developed.  This  development  should 
consider  trade-offs  between  restart  capability  and  inter¬ 
active  programming  costs. 

Additional  tasking  should  address  development  of  a 
reports  manual  for  each  model.  Such  a  model  would  define 
the  model  parameters  used  in  generating  reports.  This 
would  add  accuracy  to  interpretations,  and  encourage 
more  upper  management  use  of  DOTS . 


The  current  configuration  at  FTC,  Norfolk  has  the 
TPC  acting  as  system  analyst  who  serves  as  "link"  or 
"buffer"  between  command  executives  and  instructors  and 
system  operators.  Access  to  DOTS  is  therefore  available 
at  all  levels  through  these  analysts.  This  procedure 
appears  to  make  for  better  translation  of  problem  state¬ 
ments  and  may  save  processing  costs,  but  little  docu¬ 
mentation  or  experience  exists  to  substantiate  these 
effects  or  that  more  direct  access  from  upper  and  lower 
levels  would  result  in  more  use/abuse. 

Training  for  DOTS  should  occur  as  soon  as  possible 
after  reporting  aboard  (at  least  at  local  command  level) , 
not  only  so  that  models  will  be  effective  over  a  longer 
period,  but  also  to  provide  trainees  with  a  better 
understanding  of  the  total  training  command  system  early, 
and  therefore  reduce  experience  required  to  effectively 
manage  within  the  TRACOM.  (This  is  spin-off  from  models) . 

Update  Requirements 

It  was  difficult  to  evaluate  this  area  because 
sufficient  new  problems  were  not  used  in  the  T&E.  Program 
(model)  updating/modification  would  probably  be  rare  for 
routine  (local  command)  report-generation.  A  more 
extensive  program  update  requirement  would  probably  exist 
at  upper  management  levels  for  resolving  projected 
("what-if")  problems.  Updating  of  master  data  base  and 
configuration  control  of  programs  (models)  should  be 
centrally  managed,  and  should  be  specified  prior  to 
completion  of  R&D  phase.  Central  management  currently 
resides  with  system  sponsor  (CNET  N-5) ,  and  should  remain 
at  this  level  for  short-term  (2-3  year  R&D  phase) .  Long¬ 
term  (operational)  management  will  reside  with  system 
manager  who  will  provide  operations/program  support. 

The  system  manager  may  eventually  be  an  activity  like 
NETISA,  but  as  stated  above,  this  should  be  determined 
near  end  of  R&D  phase  and  by  the  system  sponsor. 

The  capability  exists  for  data  base/model  changes  at 
the  local  command  ("school  house")  level  via  scratch 
discs,  although  these  modifications  will  probably  not  be 
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frequent.  These  modifications  can  be  made  on  the  scratch 
discs  without  endangering  master  systems.  This  scratch 
capability  permits  resolution  of  problems  which  are 
specific  to  local  commands.  These  modifications  could 
involve  additional  parameters  such  as  assigning  course 
priorities,  accounting  for  deployment  status  of  local 
ships,  flow  patterns  between  facilities,  or  optimization 
of  other  resource  variables  (such  as  "scientists/techni¬ 
cians"  and  "projects'1  instead  of  "instructors"  and 
"students"  for  the  CNETS) .  These  modifications  and 
extraneous  applications  should  be  covered  and  documented 
in  future  tasking.  In  addition,  the  feasibility  of  these 
modifications/applications  should  be  addressed. 

Update/modification  requirements  in  the  case  of  upper 
management  may  center  around  problems  involved  in  inte¬ 
grating  DOTS  with  other  information  systems.  A  serious 
effort  should  be  undertaken  to  determine  complementary 
functions  among  these  information  systems  if  redundancies 
are  to  be  minimized  and  efficient  integration  and  upper 
management  usage  are  to  be  attained.  This  effort  is  not 
part  of  the  current  DOTS  mission  or  tasking. 

Although  configuration  management  will  be  centralized, 
major  training  centers  should  have  terminals  which  will 
provide  the  scratch  capabilities  previously  mentioned. 
TRAPAC  representatives  states  that  terminals  may  be 
clustered  around  major  training  centers  for  providing 
common  and  specific  data,  and  for  resolving  local 
problems.  Common  data  from  all  centers  would  then  be 
provided  to  the  system  (configuration)  manager  (preferably 
on-line  via  batched  magnetic  tape  or  disc  input  instead 
of  mailing  flat-paper  or  punched  cards).  Common  data 
base  updating  should  be  routine  and  periodic  (about  once 
a  week)  for  most  data  elements.  This  will  permit 
effective  use  of  DOTS  by  insuring  that  perishable  data 
is  fairly  recent  and  that  the  data  are  uniform  (for  time) 
across  commands.  Local  commands  and  centers  would  have 
access  to  lateral  (other  local)  data  bases  (or  to  command 
specific  data  in  master  data  base)  on  an  "as  required" 
bas is . 
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A  future  taskrng  requirement  exists  to  develop  a 
planning  document  showing  geographical/functional 
distribution  of  finalized  DOTS  hardware  system,  as  well 
as  software  capabilities  and  software  central  sources. 

A  functional  flow  chart  should  also  be  developed  for 
the  TRACOM  showing  scope  and  frequency  of  decision 
making  information/problem  solving  required  at  all 
levels,  including  specific  examples  of  problems  and 
methods/flow  used  to  resolve  these  problems.  A 
finalized  DOTS  may  include  expanded  data  base  (more 
parameters)  and  modified  models  if  this  functional  flow 
analysis  shows  such  a  requirement  exists.  This  infor¬ 
mation  may  include  additional  courses  common  to  many 
schools  and  centers,  as  well  as  additional  input  from 
other  information  systems  such  as  NITRAS . 

Basic  data  collection/verification/input  to  the 
master  data  base  would  be  responsibilities  of  local 
commands  (centers).  These  procedures  are  more  acceptable 
than  with  other  current  information  because  local 
commands  would  have  direct  access  to  central  system. 

These  commands  would  also  have  to  manage  problem¬ 
solving  requirements  unique  to  the  command  or  center. 

This  responsibility,  however,  is  viewed  as  a  benefit 
to  local  users  inasrm  ch  as  it  permits  resolution  of  local 
problems  without  upper  management  intervention.  Such  a 
benefit  should  improve  the  reliability/validity  of  master 
data  base  input  because  the  local  commands  would  require 
reliable/valid  data  to  resolve  local  problems.  The  final 
data  management  system  may  be  similar  to  that  presently 
used  for  CMI . 

The  T&E  Team  did  not  think  the  programs  and  data 
base  associated  with  the  SCRR&TPF  models  would  be 
frequently  updated  in  the  operational  phase.  Although 
the  data  base  of  the  ETE*  model  might  require  frequent 
update,  this  should  not  present  much  of  a  management 
problem  because  the  master  data  base  would  not  be  involved 
(only  scratch  disc) .  The  ETE  model  per  se  would  probably 


*lt  was  strongly  indicated  that  the  name  of  the  ETE 
model  should  be  changed  to  the  more  appropriate  title, 
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nol  Lx.1  i  roqui'n1  ly  upd.  i  L  o*  1 .  Updating  ol  .my  mod  1  would 
probably  be  done  within  Lwu  weeks  for  most  single- 
factor  mod i 1 icat ions .  This  timeframe  depends  on  the 
interaction  of  these  factors  with  others  in  the  model, 
however . 

The  SCRR  Si  TPF  models  appear  to  be  useful  at  all 
TRACOM  levels,  while  ETE  -  more  useful  at  the  local 
("school  house")  level.  Approval  of  permanent  ETE  model 
changes  would  have  to  be  coordinated  at  system  manager 
level.  The  Tap,  Team  believed  that  neither  master  data 
base  nor  program  modifications  would  be  difficult  if 
proper  justification  for  making  these  modifications 
could  be  demonstrated  (through  findings  of  R&D  or 
training  effectiveness  evaluations) . 

Inasmuch  as  DOTS  is  primarily  a  projection/management 
system  which  provides  information  for  "what  if"  questions, 
its  use  as  an  information/report-generating  system  would 
be  redundant,  and  would  represent  gross  underusage  of 
the  system.  Information/report  generation  would  not  be 
adequate  justification  for  use  at  local  or  upper  manage¬ 
ment  levels.  DOTS  is  not  primarily  an  information 
collecting/collating  system,  but  instead  provides  optimi¬ 
zation  of  information  output  from  other  systems  (i.  e., 
NITRAS).  The  three  DOTS  models  evaluated  should  be  viewed 
as  subsystems  within  a  TRACOM  MIS. 

Response  Times 

If  used  as  an  information  gathering  system  only,  DOTS 
would  not  substantially  improve  on  response  times 
associated  with  other  available  systems.  Response  times 
for  actual  manipulations  are  associated  directly  with 
frequency  of  the  problem-solving  requirement. 

Output  Form 

The  needs  to  be  standardized,  but  the  reformatting 
requirement  should  be  easily  accomplished  as  the  system 
is  refined. 
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Miscellaneous 


It  was  determined  that  the  restrictions  (sole  source) 
placed  on  DOTS  by  the  proprietary  RAMIS  DBMS  were  not 
acceptable,  and  that  although  an  interactive  programming 
capability  was  necessary  for  DOTS,  the  RAMIS  system 
should  either  be  purchased  by  Navy  or  a  similar  system 
should  be  obtained  permanently  from  another  source. 
Additional  tasking  should  identify  these  other  sources. 
These  sources  should  have  the  following  characteristics: 

1.  A  data  base  management  capability. 

2.  An  interactive  programming  capability. 

3.  Report-generating  capability. 

4.  Capability  to  program  in  higher  language  as  per 
current  DOD/DON  requirements. 


Nature  of  the  Data  Utilized  by  the  Models 


The  data  appears  valid.  There  may  be  a  requirement 
to  add  some  method  for  validating  relationships  of  data, 
etc.,  beyond  standard  format  checks. 

A  capability  is  needed  to  easily  add/modify  data 
elements.  This  may  be  satisfied  through  the  continued 
use  of  a  data  base  management:  system  (DBMS)  .  A  DBMS 
should  be  a  mandatory  feature  of  the  system,  if  ease  of 
use  is  to  be  retained. 

The  TPF  model's  treatment  of  backlog  does  not  properly 
represent  the  phenomenon  nor  is  it  related  in  any  pre¬ 
dictable  way  to  "Demand”.  What  are  the  factors  involved 
and  the  difficulties  in  getting  these  data?  There  may  be 
a  requirement  for  data  from  BUPERS ,  etc. 

Some  sort  of  training  reservation  system  would  be 
needed  to  gather  better  demand  and  backlog  data.  The 
T&E  should  be  expanded  to  examine  viable  alternatives 
(one  exists  at  COMTRAPAC) . 

"Bureau  capacity"  and  "Demand"  variables  used  in  the 
models  are  of  questionable  utility  for  some  locations. 
Retain  this  as  "Capacity  A"  and  "Capacity  B"  for  local 
use  vice  a  specific  use  throughout  the  system. 

The  "J"  number  is  no  longer  used. 

To  encourage  maximum  validity,  data  elements  stan¬ 
dardization  should  be  accomplished  in  accordance  with 
current  instructions. 

Some  additional  data  elements  may  become  necessary. 
Caution  should  be  exercised  in  view  of  the  costs  asso¬ 
ciated  with  gathering,  validating,  etc.  The  variables 
and  constraints  in  the  model  can  represent  other  elements 


or  resources  other  than  those  described  in  the  documenta¬ 
tion.  The  various  levels  of  application,  i.  e.,  CNET, 
CNTT,  SSC,  FTC,  COMTRALANT/PAC  may  drive  the  need  for 
additional  elements. 

Ensuring  accurate  data  will  ultimately  require  a 
system  to  provide  for  local  validation  of  data  as  part 
of  the  submittal  system.  This  would  be  part  of  the 
feedback  process,  if  the  originator  has  data  available, 
and  uses  it,  the  validity  of  the  data  should  be  greater. 

Conclusion :  The  core  of  the  data  base  is  NITRAS  data 
which,  as  "raw  data, "  is  relatively  valid,  reliable  and 
accurate,  (except  as  noted  above) . 


Data  Availability 

The  data  are  presently  available,  largely  without 
additional  co3t.  Perhaps  all  the  data  is  not  available 
at  CNTT  activities.  Perhaps  less  detailed  data  would  be 
required  if  models  were  utilized  at  higher  management 
levels.  Summaries  should  be  available  (or  at  least  the 
capability)  at  higher  echelons.  A  belief  exists  that 
data  entry  should  remain  at  the  activity  level.  This 
would  assure  accuracy  of  submission  and  avoid  possible 
over  generalization  of  problems  at  higher  echelons. 

Data  Accessibility 

The  data  is  not  presently  either  easily  or  regularly 
accessible.  (I.  e..  Programs  must  be  written  to  extract 
the  available  basic  data.)  Some  sort  of  on-line  system 
for  gathering  and  reorganizing  data  would  be  ideal. 
Accessibility  by  users  should  be  controlled  in  order 
that  the  system  user  has  access  to  only  his  own  data. 
Other  data  would  be  available  on  an  as  required  basis. 
Right  to  privacy  considerations  will  of  course  apply  if 
future  use  is  made  of  the  presently  dormant  student  pro¬ 
file  data. 

Data  Management 

Data  base  is  manipulatable  in  a  very  impressive  way, 
however .  th^  system  for  accomplishing  this  manipulation 


is  not  Navy  owned  and  is  equipment  specific.  The  data 
is  effectively  centrally  managed,  through  the  NITRAS 
reporting  system.  The  T&E  team  concluded  that  even  with 
all  the  existing  problems  in  NITRAS,  further  improvement 
and  corrective  actions  to  improve  the  existing  system 
should  be  fully  supported  throughout  the  EDTRACOM. 

Future  Data  Problems 

Essentially  covered  under  previous  discussions. 

Other  Discussions 

The  usability  of  data  to  be  accessed  by  the  DOTS 
family  of  models  relates  directly  to  the  NITRAS /NAVTI S 
problems;  such  problems  as  validity,  accessibility, 
availability,  etc.,  hinge  on  NITRAS  and  other  systems 
(example,  SHORSTAMPS  SHOROC,  etc.)  either  operational  or 
under  development  of  being  implement ated .  The  DOT  models 
must  be  viewed  as  a  capability/application  consistent  with 
the  total  Training  Command  MIS  and  not  as  a  separate 
system.  From  specific  and  general  comments  by  attendees 
to  this  T&E.  a  strong  display  of  feeling  was  evident  that 
the  EDTRA  Command  is  in  dire  need  of  management  in  the 
area  of  systems  development  and  coordination.  Specifically 
there  is  the  current  problem  of  a  lack  of  coordination  of 
systems  development  and  interfacing.  There  is  a  require¬ 
ment  for  an  overall  information  systems  management/ 
development  function  that  is  independent  from  the  opera¬ 
tional  administration  of  the  ADP  facilities  in  the 
EDTRACOM.  Some  form  of  information  systems  study/develop¬ 
ment  function  has  to  exist  that  addresses  the  needs  of 
all  levels  of  the  Command,  not  just  CNET.  This  could  be 
implemented  through  the  use  of  a  central  coordinating 
function  at  CNET  not  associated  with  the  present  ADP 
administration  function,  and  staffed  with  representatives 
(NON-ADP)  from  various  functions  and  levels  throughout 
the  EDTRACOM.  Serious  doubt  has  been  expressed  about  most 
systems  under  development  in  that  there  are  varing  overlaps 
of  objectives,  redundanacy  in  data,  and  duplication  in 
reports  generated  when  all  systems  are  viewed  jointly. 
Additionally,  the  strong  possibility  exists  that  by  and 
large  the  systems  now  under  development  or  proposed  are 


essentially  designed  to  serve  only  the  CNET  level  of 
need  and  are  of  little  or  no  utility  to  the  operational 
levels  and  functional  levels  of  the  EDTRACOM  except  in 
several  unique  cases.  Very  littD.e  effort  appears  to  be 
directed  towards  the  development  systems  like  DOTS  which 
have  the  inherent  capacity  to  become  viable  planning  tools 
(i.  e.,  for  forecasting,  modeling,  etc.)  or  as  tools  for 
evaluating  various  approaches  to  resource  utilization. 


Organizational  Implementation  ('Fourth  discussion  Area) 

Levels  of  Application 

Discussion ;  Specific  level (s)  and  degree  of  appli¬ 
cation  of  the  DOTS  models  cannot  be  defined  at  this  time. 

The  considerations  which  preclude  such  definition  are 
essentially  the  lack  of  costing  information  and  lack  of 
experience  based  on  application  beyond  FTC,  Norfolk. 
Discussions  regarding  systems  costs  and  economy  of  opera¬ 
tion  centered,  in  the  main,  on  whether  or  not  schoolhouse  - 
level  implementation  could  ever  be  shown  to  be  economi¬ 
cally  feasible  or  even  justifiable.  This  argument  centered 
on  the  premise  that  the  unavailability  of  precise  costing 
data  notwithstanding,  the  thrust  of  implementation  should 
be  toward  the  functional  level,  in  that  only  at  this  level 
would  the  magnitude  and  complexity  of  problems  warrant 
the  capability  to  apply  modeling  techniques. 

Counter  -  arguments  stated  the  premise  that  costs 
would  in  all  likelihood  be  minimal  to  the  point  that  they 
would  not  be  the  primary  deterrent  to  systems  implementa¬ 
tion;  and  further,  that  there  were  many  ways  in  which  the 
models  could  (should)  be  used  at  the  activity  level.  That 
the  amount  of  usage  and  attendant  benefits  to  be  derived 
from  such  use  is  limited  only  by  the  imagination  and 
resourcefulness  of  the  activity  itself. 

Conclus ion :  Throughout  the  T&E  we  have  been  hampered 
by  a  lack  of  costing  data.  We  are  unable  to  predict  with 
any  degree  of  validity  what  this  capability  might  cost  in 
any  of  its  terms  (hardware,  software,  personnel,  tele¬ 
communications,  site  prep)  much  less  as  a  whole.  This  is 
not  a  criticism,  in  that  derivation  of  systems  costs  was 
not  a  goal  of  the  effort  to  this  point,  but  rather  a 
statement  of  fact.  This  fact  does  serve  to  highlight  the 
other  issue  however  in  that  while  activity  level  usage 
might  be  great  in  terms  of  numbers  of  interactions,  does 
the  utility  warrant  the  investment?  We  cannot  say  at  this 
time.  The  most  we  can  say  is  that  if  continued  evaluation 
is  directed,  that  a  portion  of  that  evaluation  must  address 
the  economics  of  the  system  auu  that  the  economic  evaluation 
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neiationsnips  or  reoaexs  co  existing  roncxes  ana 
Structures : 


Discussion :  The  potential  for  contribution  by  the 
DOTS  models  in  the  overall  planning  area  is  its  most 
significant  feature.  The  need  for  accurate,  timely 
planning  is  well  recognized  and  there  are  several  systems 
now  operational  or  under  operational  development  and 
implementation  which  purport  to  provide  this  capability 
(none  appear  totally  satisfactory  be  it  a  function  of 
scope  of  concept  or  of  performance) .  DOTS  is  envisioned 
as  a  possible  device  to  effect  the  changes  required  in 
these  various  systems  to  provide  a  truly  reliable  and 
accurate  MIS  with  the  added  ability  to  model.  This 
added  capability  should  provide,  through  linear  program¬ 
ming  and  optimization  techniques,  training  managers  a 
most  valuable  and  highly  responsive  tool  to  aid  in  the 
decision  making  process.  (Costs  notwithstanding).  Con¬ 
siderations  concerning  receptivity,  resistance  and 
incentives  centered  about  the  concerns  of  unnecessary 
higher  echelon  involvement  with  everyday  problems  most 
appropriately  addressed  by  the  cognizant  activity,  and 
the  ability  of  the  system  to  provide  real  time  infor¬ 
mation  and  assistance  to  the  user  ...  as  contrasted  to 
present  systems  wherein  the  activity  realizes  no  return 
on  their  input  investment.  Even  with  the  concern 
expressed  above,  it  was  the  consensus  that  given  further 
evaluation  the  activities  would  be  receptive  to  using 
the  models.  Implementation  support  was  considered 
generally  excellent. 

Conclusion :  If  the  DOTS  concept  did  nothing  more 

than  precipitate  an  investigation  and  realignment  of  the 
current  morass  of  Training  Command  MIS (s)  it  would  be  of 
value.  Given  that  it  will  provide  capabilities  to  model 
and  simulate  and  therefore  project  as  well,  jt  is 
potentially  of  great  value. 

The  central  point  is  that  DOTS  is  not  to  be  another 
system  separate  and  distinct  from  those  already  in 


being.  It  must  not  be  competitive.  It  must  be  comple¬ 
mentary  to  a  total  MIS  established  from  the  myriad  of 
(unintegrated)  systems  that  now  exists.  Similarly,  DOTS 
(as  a  function  of  the  total  Training  Command  MIS)  must 
interface  with  those  systems  external  to  the  Training 
Command,  such  as  SHORSTAMPS,  that  will  impact  on  the 
Command . 

While  the  ETE  and  highly  conceptual  ETAM  models 
would  appear  to  have  the  least  impact  external  to  the 
Command,  they  might  have  the  greatest  impact  within. 

The  magnitude  of  the  individualization  effort  within  CNTT 
alone,  and  the  projected  savings  from  this  effort,  would 
support  continued  R  and  D  of  these  models. 


or  could  present  staff  learn  to  operate  models  on  vhe 
results?  The  group's  concensus  v;as  that  no  additional 
personnel  would  be  needed.  That  while  some  training 
could  be  necessary,  present  staff  can  learn  to  use  the 
models/results,  inmost  cases  the  wider  familiarity  with 
the  program  within  the  command  the  better.  Resources 
would  have  to  be  identified  to  accomplish  configuration 
management  (hardware  and  software)  and  necessary  coordi¬ 
nation/instruction  among  commands  using  DOTS  and  between 
DOTS  and  other  systems.  Most  likely  this  would  be  accom 
plished  best  by  some  present  facility  with  the  necessary 
expertise  and  experience  in  this  area. 

How  much  additional  staff  time  would  be  required  to 
operate,  maintain  and  utilize  the  models  and  their  data 
base?  This  variable  is  largely  unknown  at  this  point. 

It  is  however  directly  related  to  use  which  is  a  function 
of  the  acceptance  and  utility  of  the  models  themselves. 

Hardware 

What  hardware  acquisitions  would  typically  have  to 
be  made?  Many  existing  hardware  configurations  now  in 
place  at  potential  user  sites  could  be  used  for  DOTS. 

The  minimum  requirements  (i.  e.,  keyboards/printers) 
would  suffice  in  many  cases.  CRTS  and  other  more  sophis¬ 
ticated  peripherals  would  not  be  necessary  in  many 
locations,  in  a  good  many  cases,  communicating  magnetic 
card  typewriters  could  fulfill  all  the  hardware  require¬ 
ments  for  access  to  the  DOTS  system,  without  significant 
impact  on  present  use  of  this  equipment.  Many  training 
activities  already  possess  such  equipment.  In  a  non¬ 
time  share  application  of  these  systems,  more  significant 
hardware  acquisitions  would  be  necessary  unless  surplus 
capacity  on  Government  controlled  equipment  was  available 
in  either  case  program  update  capability  via  magnetic 


tape  and  sufficient  disc  storage  capacity  are  required 
as  is  some  Data  Base  Management/Rapid  Access  Report 
Generation  System.  For  these  reasons  and  others  addressed 
above,  DOTS  ■will  be  most  efficiently  utilized  as  a 
commercial  time  share  system. 

what  is  the  Approximate  cost?  Present  erst  data  are 
based  on  general  estimates  provided  by  IBM.  Certain 
general  conclusions  are  possible.  However,  detailed  cost 
analysis  is  required  prior  to  submissions  of  an  ADS  plan. 
This  should  be  undertaken  with  qualified  personnel  at  a 
time  subsequent  to  the  T&E.  Cost  will  be  dependent  on  a 
variety  of  factors  including  but  not  limited  to: 

Hardware  configuration 

Processing  time 

Necessity  for  changes 

Number  of  ultimate  users 

Future  expansion 

Mode  of  operation 

Relation  with  other  systems 

Extent  of  data  collection/reduction  etc. 

Per  unit  cost  will  be  amoratized  better  with  greater  in¬ 
cidence  of  use,  and  total  cost  can  be  reduced  if  hardware 
requirements  are  kept  to  a  minimum. 

Given  findings  in  first  and  fourth  discussion  areas, 
what  might  be  a  "ball  park"  cost  per  incidence  of  usage 
ratio .  $20K  a  year  estimate  by  IBM  seems  excessive  if 

hardware  is  largely  in  place. 

Method  of  Operation 


Time  Sharing 

For  the  short  term  (2-3  years) 


Commercial  time  -  sharing  appears  to  be  the  only  feasible 
alternative  while  the  project  remains  in  ADO  form. 


Other  Discussion 


A  general  agreement  evolved  during  discussion  of 
user  requirements  to  the  effect  that  expansion  of  the 
prototypical  modeling  system  to  TRAPAC  and  CNTT  activi¬ 
ties  is  justified.  Such  expansion  would  address  a  wider 
variety  and  greater  frequency  of  training  decision  making 
conditions,  explore  the  interfaces  necessary  to  complement 
existing  and  developing  systems,  expand  the  data  base, 
increase  familiarity  with  modeling  as  a  technique, 
potentially  improve  collection  of  related  data,  and  provide 
better  indications  of  system  potential  chan  would  an 
extensive  effort  to  go  operational  at  this  time.  Each 
problem  area  presented  to  the  models  for  information  gene¬ 
ration  in  support  of  decisions  inherent  in  the  problem 
should  be  carefully  documented  during  this  phase  for  later 
evaluation  with  regard  to  system  costs.  Certain  commands 
expressed  concern  about  spending  that  much  money  over  a 
period  of  time  without  guarantees  that  continued  access 
to  the  system  would  be  provided.  Future  phases  of  DOTS 
should  test  a  variety  of  operational  set-ups  under  minimum 
hardware/support  conditions,  under  a  cluster  or  network 
configuration,  etc. 

Conclusion 


This  whole  area  is  essentially  a  moot  point  unless  and 
until  an  ADS  plan  is  necessary.  This  requirements  seems 
to  be  about  1-2  years  away.  If  this  system  progresses  to 
a  recommendation  for  operational  implementation,  system 
specifics  should  be  left  to  further  determination  during 
the  procurement  process  and  be  done  in  such  a  way  as  to 
ensure  investigation  of  all  possible  configurations  with 
attendant  costs.  (See  Appendix  H  for  pertinent 
instructions . ) 


User  T&E  Conclusions  and  Recommendations 


The  following  conclusion*  and  recommendations  were  made 
by  the  User  T&E  members: 

•  DOTS  should  be  continued  as  an  ADO. 

.  The  thrust  of  DOTS  should  be  modified  to  include  the 
investigation  of  applications  at  higher  levels  (i.  e., 
beyond  FTC)  within  CNET. 

.  The  ETAM  project,  while  not  addressed  specifically 
in  the  T&E,  warrants  support  in  view  of  information 
provided  to  date. 

.  The  next  phase  of  DOTS  should  include  field  test (s) 
of  the  developed  models,  under  operational  conditions. 

.  Investigation  of  the  potential  management  improvements 
embodied  by  DOTS  underscores  the  need  to  integrate  and 
synthesize  the  plethora  of  management  support  systems 
internal  to  or  impacting  upon  CNET  upper  level  manage¬ 
ment  (e.  g.,  STREPS,  NITRAS,  MAPMIS,  RACS,  CENTRA,  CMI , 
NATIS,  sqcAeas-trs,  NCFA,  SHORSTAMPS)  . 

.Upon  the  formulation  of  a  draft  T&E  report,  a  future 
meeting  of  CNET  function  representatives  should  be 
held  in  order  to  finalize  the  T&E.  An  end  product  of 
this  meeting  should  be  a  specific  statement  of  commit¬ 
ment  from  the  various  CNET  functions  regarding  the 
future  of  the  DOTS  project.  This  should  include  organi 
zational  support  and  participation  in  the  project. 


MANAGFMJWT  SUMMARY 


Introduction 

This  teot  and  evaluation  dealt  with  three  computer  based  models 
that  were  selected,  developed,  and  validated  for  application  at  the 
Fleet  Training  Center,  Norfolk,  Va.  The  total  cost  of  this  segment 
of  the  DOTS  program  is  estimated  at  about  $780  K,  The  following 
outcomes  have  resulted  from  this  expenditure: 

(1)  A  determination  of  the  feasibility  of  applying  management 
science  techniques  to  improve  Navy  training  management. 

(2)  Three  working  models  were  developed  and  documented  that 
have  potential  as  tools  to  address  a  broad  array  of  training 
management  problems  and  have  specific  applicability  to 
fleet  training. 

(3)  A  data  base  framework  was  developed  which  is  approximately 
90%  consistent  with  data  presently  being  reported  within 
CNET  (i.  e.  ,  NITP_AS)  and  within  which  data  have  been 
collected  and  refined  for  FTC,  Norfolk.  This  provides  a 
potential  means  for  facilitating  present  and  future  reporting 
of  command  training  information  to  NITRAS  or  other  CNET 
management  information  systems. 

(4)  Application  of  a  standard  data  base  management  system 
(RAMIS)  to  the  above  data  base. 

(5)  Establishment  of  a  working  management  information  center 
at.  FTC,  Norfolk  which  utilized  the  models,  data  base,  tele¬ 
communications  equipment,  and  trained  operators. 

(6)  Extensive  documentation  of  the  models,  their  rationale,  and 
operation. 

(7)  Exposure,  of  a  considerable  number  of  key  Navy  training 
managers  to  the  possibilities  for  applying  management 
science  techniques  in  the  solution  of  their  problems. 
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The  780K  referred  to  above  rcprcuontu  approximately  half  of  the 
total  expenditures  for  the  DOTS  project  to  date.  To  avoid  misinterpre¬ 
tation  of  the  T&E  it  is  important  to  emphasise  the  distinction  between 
nn  ovnloatlnn  of  n  togmont  o£  the  project,  as  summarised  below,  and 
an  evaluation  of  the  project  as  a  whole.  The  following  provides  an 
indication  of  DOTS  developments  which  were  not  dealt  with  directly 
by  this  T&E: 

•  Comprehensive  functional  study  of  NAVEDTRACOM 

•  Strategic  assumptions  for  the  1980  decade 

•  Delineation  of  technological  gaps  in  the  training  system 

•  Initial  test  of  the  feasibility  of  using  simulation 

•  Development  of  the  Educational  Technology  Assessment 
Model  (ETAM) 

•  Naval  Training  Program  Requirements  Study  and  Model 
(Dahlgren) 

•  Study  on  application  of  decision  theory  to  CNET 

•  Audio-Visual  aids  for  orienting  Navy  training  managers 
to  the  nature  and  use  of  management  science  techniques 

Feasibility  Assessment 


It  is  the  conclusion  of  this  analyst  that  the  DOTS  models  have 
established  beyond  reasonable  doubt  the  feasibility  of  applying  modeling 
techniques  to  Navy  training  management  problems.  This  general  - 
finding  is  a  composite  of  the  elements  of  technical,  operational,  and 
economic  feasibility  ns  discussed  below.  The  three  models  can  be 
dichotomized.  Most  importantly  there  are  the  System  Capabilities/ 
Resources  Requirements  Model  (SCRR)  and  the  Training  Process 
Flow  Model  (TPF),  which  arc  resource  allocation/process  flow  type 
models.  Secondarily  there  is  the  Educational  Technology  Evaluation 
Model  (ETE)  which  is  a  simulation  based  course  design  tool  for 
Independent  Learning  Systems  (JLS). 

Ted* i ii  ca1_  F <  ;mi bi  1  i t  y 

The  SChl^nnd  T_PF  modulo  arc  very  complementary  in  nature 
and  share  n  common  level  of  application.  It  is  not  surprising  at  all 
ili-i I  those  two  model  type;,  were  selected  for  development.  It  is  fair 
to  say  that  it  was  not  mu  or  navy  to  perform  the  extensive  analysis 
of  Eh. i so  I  male  (lie  <|.  termination  that  n  resource  allocation  model 

and  a  pro*  fl.iv.  . .  1  should  he  among  the  forms  first  attempted 

foi  CNtri  .  I’Ii-.m*  1,  indeed  ss.if.  a  noeenunry  f.d:op  in  identifying  where 
mu  h  i  irnl'li;  woo  hi  In  best  applies  and  fur  identifying  the  greater 
.  y  if  io  t-m-i  r*M,iui  nf  in  whb.h  they  mast  operate. 
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Certainly,  the  models  are  .not  uniform  in  their  current  level  of 
technical  development.  For  example,  the  Training  Process  Flow  Model 
(TPF)  failed  to  reach  some  of  its  envisioned  goals  in  that  statistical 
analysis  of  student  profile  data  failed  to  produce  correlations  of 
significance  enough  to  warrant  inclusion  in  the  model.  As  a  result, 
the  TPF  presently  does  not  have  the  capability  to  relate  the  school- 
house  process  to  the  characteristics  of  incoming  students.  However, 
while  failing  to  establish  the  feasibility  of  the  statistical  approach,  it 
hardly  proved  its  infeasibility.  In  fact,  the  nature  of  the  analysis 
conducted  begs  alternative  approaches  which  might  be  taken  in  the 
future.  In  its  current  nonstatistical  form,  the  TPF  has  shown  that  it 
can  he  a  useful  tool  in  the  hands  of  training  managers  by  taking  full 
advantage  of  presently  available  data  and  certain  operationally  utilized, 
if  still  ill-defined,  planning  factors  extant  in  the  current  training 
environment.  Additionally,  the  TPF  seems  to  have  carried  the  brunt 
of  initial  data  base  development  for  the  two  models. 

The  SCRR  model  demonstrates  the  technical  feasibility  of  providing 
training  center  managers  with  more  accurate  and  multi -dimensional 
assessments  of  their  training  capabilities.  Like  the  TPF,  it  takes 
advantage  of  existing  planning  factors  such  as  the  student/instructor 
ratios  which  reflect  basic  requirements  formulas  in  order  to  determine 
organizational  requirements  on  the  basis  of  the  interaction  of  resources 
and  capabilities.  When  more  precise  planning  factors  become  available 
in  the  future  (e,  g.  ,  through  the  SHORSTAMPS  program),  they  can 
be  easily  assimilated  into  the  model. 

The  constant  returns  to  scale  assumption  of  the  linear  program 
does  not  appear  to  be  a  limiting  factor  in  its  application  within  the 
training  environment.  Though  the  "step"  function  is  perhaps  most 
descriptive  of  the  relationship  between  training  resources  and  output, 
there  is  no  basic  inconsistency  with  the  LP  approach,  especially  in 
view  of  integer  and  mixed  integer  formulation  options. 

The  SCRR,  in  a  sense,  is  an  automation  of  what  is  currently 
being  done  mentally  by  instructors  and  training  managers.  This  is 
viewed  as  a  plus.  Using  the  SCRR,  within  the  course  of  an  hour,  it 
might  bo  possible  to  develop  as  many  resource  allocation  alternatives 
as  a  training  manager  could  develop  in  two  weeks.  This  advantage,  of 
course,  would  be  multiplied  many  times  over  considering  frequency  of 
occurrence.  Add  to  this  the  capability  for  optimizing  across  resource 
allocation  alternatives  and  one  can  see  that  the  SCRR  presents  the 
potential  for  some  very  real  and  immediate  payoffs  for  the  training 
m;i  rat  g  >•  r. 
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The  SCRR  and  TPF  models  wore  developed  in  a  fleet  training 
environment.  Here  the  demand  is  characteristically  unprogrammed. 

It  is  reasonable  to  believe  that  in  other  Navy  training  sectors  where 
demand  is  much  more  programmable  (c.  g.  ,  technical  training),  the 
potential  for  effective,  model  development  is  at  least  as  great  if  not 
greater  than  has  been  demonstrated  for  the  fleet  environment. 

In  contrast  to  the  SCRR  and  TPF,  the  ETE  model  focuses  on  the 
course  -  its  design  and,  to  some  extent,  its  management.  It  is  quite 
independent  from  the  other  models  in  both  content  and  context.  It  is 
not  nearly  as  organizationally  limited  as  the  other  two  and  theoretically 
lias  applicability  wherever  IRS  courses  are  designed,  modified,  or 
where  a  significant  number  of  such  courses  must  be  managed  given 
some  commonality  of  resource  requirements. 


No  particular  faults  could  be  found  with  the  composition  of  the  ETE. 
Its  primary  problem  is  that  while  none  of  the  models  were  strictly 
field  tested,  the  ETE  was  the  least  tested  of  the  lot.  In  the  absence  of 
IRS  at  FTC,  Norfolk,  the  model  was  validated  against  an  actual  JLS 
course  at  Corry  Station.  The  results  of  this  validation,  while  thoroughly 
documented,  do  not  provide  insights  to  how  useful  the  model  can  be  to 
actual,  users.  Besides  being  a  very  limited  test  (i.  e.  ,  one  school), 
the  validation  was  against  the  results  of  another  simulation  model  (in 
this  case,  a  special  purpose  type).  This  resulted  in  substantia]  proof 
that  the  ETE,  despite  its  generality,  could  perform  in  superior  fashion 
to  a  system  specific  model  in  a  given  case.  It  does  not  tell  anything 
about  whether  there  will  be  much  opportunity  for  use  of  such  a  model 
given  the  nature  of  the  EDTRACOM's  operations.  A  more  meaningful 
test  of  the  appropriateness  of  the  ETE  to  current  or  future  Navy 
training  problems  was  apparently  limited  by  the  unavailability  of 
appropriate  subject  courses. 

O  pc  rati  onal  Fca  nihility 


Data.  The  SCRR  and  TPF  models  were  developed  to  be  consistent 
with  data  currently  being  collected.  Inputs  to  the  NITRAS  data  base 
(which  arc  supplied  to  CNHT)  were  used  to  comprise  a  common  data 
base  for  the  two  models.  Certain  additional  data  elements  were 
collected  directly  from  the  FTC;  however,  these  comprise  only  a 
small  portion  of  the  total  data  base.  All  of  the  data  used  by  those 
models,  are  both  avaj_]:_i_b!  e  and  accessible  within  GN.ET.  About  90% 
ol  flu:  data  must  be  collected  for  reporting  to  the  NTTRAS,  regardless 
of  model  requirements..  The  situation  is  not  just  peculiar  to  FTC, 

Norfolk,  but  j ;;  true  throughout  the  CNET  commands. 
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The  data  base  is  currently  managed  using  the  RAMIS 
software.  This  is  a  proprietary  data  base  management  system  developed 
by  Mathematica  and  widely  available  from  time- sharing  vendors. 
However,  the  interactive  (i.  e.  ,  conversational)  version  of  RAMIS  is 
only  available  from  the  time-sharing  vendor  (NCSS)  currently  being 
used  to  support  the  models.  With  RAMIS  as  a  management  device, 
the  model  data  base  is  very  satisfactory  in  its  content,  flexibility  and 
accessibility.  Supposedly,  the  NITRAS  will  have  a  similar  data  base 
management  capability  in  the  future.  However,  at  present,  that  system 
has  extensive  problems.  As  presently  setup,  the  data  base  could 
run  parrallel  to  NITRAS  until  such  a  time  as  that  system  is  fully  capable 
of  supporting  the  models.  In  fact,  from  information  revealed  at  the 
User  T&E  in  June,  it  appears  that  DOTS'  use  of  NITRAS  data  provides 
a  good  model  for  what  can  be  done  with  the  data  elements  currently 
being  reported  by  commands. 

It  is  important  to  keep  in  mind  that  the  models  currently 
require  a  data  base  management  capability  such  as  RAMIS  in  order  to 
operate.  Even  a  change  to  a  comparable  DBMS  will  require  some 
software  modifications.  On  the  otherhand,  with  the  availability  of 
data  base  management  systems  such  as  RAMIS,  it  would  have  been 
wasteful  to  have  used  project  funds  to  develop  a  similar  capability  *• 
especially  since  this  was  a.  feasibility  study. 

The  ETE  data  are  supplied  directly  by  the  user  via  a  conversa¬ 
tional  prompting  program.  The  data  required  for  the  ETE  are  fairly 
simple  in  comparison  to  those  needed  by  the  other  two  models  and  their 
supply  seems  to  present  no  operational  problem. 

Organizational  Implementation.  Extensive  consideration  was 
given  to  this  aspect  of  operational  feasibility  throughout  the  T&cE.  The 
on-site  User's  T&tE  was  especially  arranged  to  have  prospective 
CNET  users  react  and  provide  feedback  as  to  the  feasibility  of  the 
models  given  their  respective  organizational  frameworks.  The  results 
of  those  proceedings  are  presented  in  the  detailed  T&E  report.  In 
general,  members  of  the  User  T&E  team  viewed  the  models  as 
operationally  feasible  at  the  TRACOM  and  schoolhousc  level.  Additional 
feedback  later  obtained  from  COMTBALANT  and  COMTRAPAC 
reinforced  this  finding.  However,  feedback  obtained  from  CNTT 
claimed  that  the  operational,  feasibility  and  desi  rabidity  of  the  models 
had  not  boon  demonstrated  at  Norfolk.  The  essential  pitfall  of  the 
models  as  viewed  by  CNTT  is  that  they  "’nave  been  primarily  designed 
to  helj'.  that  level  of  management  least  requiring  assistance  in 
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decision-making."  (i.  c.  ,  the  school  house).  CNTT  construed  this 
view  as  being  supported  by  the  acting  Director  of  Training  at  FTC, 
Norfolk,  who  reported  to  the  T&cE  team  that  the  need  to  answer  "what 
if"  questions  was  rare  (i.  e.  ,  5  or  6  times  a  year)  at  the  FTC  level. 

(Of  course  the  Director  also  had  many  good  things  to  say  aboxit  the 
models.  He  recommended  their  continued  development  at  the  TRACOM 
level  and  hoped  that  the  FTC  could  have  continued  access  to  the  models 
on  a  periodic  basis.  ) 

Comments  on  User  T&E  Results:  While  CNTT's  point  is 
sound  in  its  premise  (i.  e.  ,  that  the  FTC  site  is  atypical  of  the  greater 
EDTRACOM),  it  has  deviated  somewhat  in  its  eonchisicn  (i.  e.  ,  that 
the  operational  feasibility  of  modeling  has  not  been  demonstrated). 
Phases  II  and  ITT  of  DOTS  have  shown  that  meaningful  and  useful  models 
can  be  constructed  for  the  benefit  of  training  management.  It  has  also 
revealed  that  the  most  effective  residence  for  such  models  in  fleet: 
training  may  be  at  the  TRACOM  level. 

In  no  way  can  this  line  of  reasoning  be  extended  to  say  that 
the.  models  thus  far  developed  are  appropriate  to  CNTT's  most  critical 
management  problems.  As  CNTT  has  implied,  the  selection  of  the  FTC 
as  the  development  cite  largely  precluded  this  possibility  from  the 
beginning  of  Phase  II. 


In  fairness  to  CNTT,  it  should  be  said  that  their  alarm  that 
someone  might  try  to  spread  these  particular  inodels  across  the 
waters  of  EDTRACOM  is  not  without  cause.  Much  of  the  DOTS  document 
lion  concerning  "Projected  operations"  of  the  models  invites  this  kind 
of  fe;>r  (c.  g.  ,  TAEG  Report  No,  12-2,  Vol  I,  Phase  11  Overview, 
pp  IV- IB  through  IV- 19).  Such  presentations  seem  to  imply  that  the 
three  models,  with  a  minimum  of  modification,  c.oul.d  be  applied  in  a 
very  systematic  manner  through  nruch  of  CNET.  This, sort  of  thinking 
was  further  embellished  by  verbal  presentations  from  IBM  which 
empha.sir.cd  that  there  would  ha.vc  to  be  fairly  significant  changes  in 
the  present  Navy  training  organisation  in  order  to  take  full  advantage 
of  the  models. 


The  models  themselves  neither  require  nor  warrant  such 
incipient  n ggra ndi '/.<  me nl .  They  are  good  examples  of  how  basic 
management  science  technologies  can  be  utilized  to  assist  in  the 
solution  of  specific  management  problems.  As  such,  they  imply  that 
establishment  of  a  program,  chartered  to  utilize  management  science 
technique:*  in  the  support  of  Navy  training  management  would  be 


well  founded.  Such  a  program  should  not  operate  on  the  basic 
assumption  that  thorc  is  a  total  integrated  solution  to  the  training 
management  problem,  but  rather  on  the  objective  of  achieving 
incremental  improvements  in  management  effectiveness  by  providing 
useful  solutions  to  important  management  problems  -  regardless  of 
their  limited  generality. 

Opportunities  for  applying  the  models  at  the  TRACOM  level 
look  exceedingly  good.  Both  COMTRALANT  and  COMTRAPAC  seem 
receptive  to  application  of  the  models  in  their  respective  organizations. 
Both  commands  have  the  necessary  equipment  for  supporting  interactive 
management  models  (TRALANT  could  use  equipment  currently  being 
leased  at  FTC  in  connection  with  DOTS).  At  TRAPAC  a  civilian 
position  which  would  have  responsibility  for  data  base  and  model 
developments  is  being  created.  No  such  position  has  been  thus  far 
identified  at  TRALANT.  It  may  be  possible  to  utilize  one  of  the 
training  managers  from  FTC,  Norfolk,  who  already  has  familiarization 
with  DOTS  in  the  model  interface  role.  It  is  essential  that  an  individual 
within  each  of  the  two  commands  has  clear  responsibilities  associated 
with  the  implementation  of  the  models. 

Ease  and  Practicality  of  Use.  This  aspect  of  the  models 
was  judged  as  satisfactory  in  the  User  T&E.  Update  requirements 
for  the  data  base  were  not  excessive.  The  basic  structure  of  the  models 
could  remain  stable.  Actual  operation  of  the  models  was  possible  with 
a  minimum  of  training.  The  interactive  nature  of  the  model  and  data 
base  programming  contributed  highly  to  the  ease  of  use.  Approximately 
two  weeks  would  be  a  reasonable  time  .for  training  an  operator  using 
basic  skill  types  presently  available  on  the  organization's  staff. 
Considerably  more  time  would  be  required  to  fully  educate  training 
managers  in  potential  uses  of  the  model  -  if  for  no  other  reason  than 
that  it  would  be  best  to  extend  such  training  over  time  thereby  allowing 
real  and  current  problems  to  serve  as  examples  in  the  training. 

Documentation.  The  present  documentation  is  satisfactory 
and  should  serve  well  in  future  utilization  of  the  models.  Though  a 
users'  guide  has  been  documented,  there  still  remains  a  need  for  an 
operator's  manual  which  fully  explains  model  options,  error  messages 
and  corrective  measures.  It  it:  understood  that  such  a  manual  is 
presently  being  prepared  by  IBM. 
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3 .  Economic  Feasibility 

.  A  'benefit/cost  analysis  was  attempted  in  TAEG  Report 
No.  12-2.  A  short  coming  is  its  assumption  that  the  models  as 
presently  configured  could  be  widely  applied  throughout  CNET.  As 
pointed  out  above,  this  premise  must  be  rejected.  There  are  other 
reasons  why  the  benefit/cost  analysis  provided  by  the  contractor  is  not 
acceptable.  These  will  be  identified  in  the  detailed  T&E  report. 

It  appears  that  a  reasonable  benefit/cost  analysis  of  the  models 
can  only  be  derived  from  a  valid  field  test.  It  must  be  established 
to  what  extent  the  models  supplant  existing  procedures  or  personnel 
and  what  additional  capabilities  are  provided  -  even  more  importantly 
what  these  capabilities  buy  in  the  way  of  greater  effectiveness. 

Numerous  attempts  were  made  during  the  T&E  to  trace  back 
from  the  application  of  the  models  to  real  problems  emanating  from 
the  FTC  and  in  turn  to  assess  the  benefits  of  supporting  the  solution 
of  those  problems.  None  of  these  attempts  resulted  in  very  useful 
information.  (At  present,  representatives  of  TRAPAC  and  TRALANT 
are  conducting  exercises  with  the  models  using  real  world  problems. 

It  is  hoped  that  these  exercises  will  result  in  more  definitive  assessments 
of  potential  model  payoffs.  ) 

Certain  rather  unavoidable  problems  contribute  towards  this 
situation.  First  it  is  difficult  to  get  managers  to  perceive,  let  along 
'use,  sophisticated  models  when  their  prime  concern  at  the  moment 
lies  in  the  basic  data  area.  Second,  certain  uses  of  the  models  harbor 
implicit  threats  to  members  of  the  organization's  staff.  For  example, 
if  the  SCRR  model  was  used  to  show  that  substantial  cuts  could  be 
made  in  the  present  staff  without  effecting  performance,  it  might 
jeopardize  getting  further  information  or  support  of  the  project  from 
that  staff. 

With  the  dearth  of  information  on  the  benefit  side  of  the. 
feasibility  question,  it  iss  best  to  focus  on  cost  factors.  Here  it  can 
be  seen  that  the  future  cost  of  developing  and  operating  the  models 
may  prove  to  be  effectively  less  titan  even  assessed  by  the  contractor 
in  his  analysis.  This  position  can  be  supported  because  of  two  con  ¬ 
current  developments  to  DOTS.  First,  the  data  base  necessary  for* 
future  model  development  jy,  in  the  main,  being  compiled  regardless 
of  DOTS.  'hints ,  Ihe  cost:  of  that  data  acquisition  and  maintenance 
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need  not  he  attributed  to  the  DOTS  models.  Secondly,  the  TRACOMS 
either  have  or  are  in  the  process  of  procuring  the  necessary  hardware 
to  support  DOTS  type  models,  again,  regardless  of  DOTS.  For 
example,  TRAPAC  already  has  in  operation  a  terminal  with  full  tele¬ 
communications  capability  from  which  the  models  have  already  been 
accessed.  Thus  it  is  doubtful  that  users  would  need  to  purchase 
equipment  that  they  would  not  have  procured  otherwise. 

The  time-sharing  cost  per  usage  of  each  model  is  well 
within  reason  (User  Test  Guide  Appendix  V,  30  May  75).  Time-sharing, 
of  course,  obviates  the  need  for  large  initial  investments  in  ADP 
hardware,  thereby  putting  the  models  on  a  largely  "pay  as  go"  basis. 

It  does  not  appear  that  additional  staff  would  be  necessary  to  support 
the  models  in  TRACOM  operation.  Modifications  should  not  prove 
extensive  for  TRACOM  application.  All  in  all,  these  factors  point 
to  a  very  low  cost  for  implementation  and  operation  in  the  projected 
TRACOM  application.  As  a  result,  it  would  not  require  very  much 
"benefit"  to  break  even  in  such  a  project. 

As  for  proving  the  financial  feasibility  of  applying  modeling 
techniques  in  general  throughout  the  EDTRACOM,  it  is  not  believed 
that  this  could  presently  be  established.  Applications  can  differ  so 
markedly  with  respect  to  cost  and  payoffs  that  any  specific- application 
hardly  provides  a  basis  for  extrapolation  of  feasibility  for  the  general 
case.  The  history  of  management  science  is  strewn  with  examples 
of  very'  good  general  models  which  never  were  implemented. 

Situational  factors  for  each  case  seem  to  override  any  generalizations 
that  might  be  made. 

Recommendations 


Recommendations  for  the  Conclus  ion  of  DOTS  (FY-76) 

1.  Initiation  of  data  collection  and  review  for  TRALANT /TRAPAC 
application  and  field  test  of  SCRR-TPF.  This  should  include  a  clear 
assessment  of  opportunities  for  model  contributions  at  these  commands. 
Each  command  should  designate  an  individual  from  its  staff  who  will 

be  the  primary  interface  point  in  model  development  and  have  a  clear 
responsibility  for  seeing  that  the  models  get  put  to  real  and  practical 
use  within  the  command  to  the  extent  possible. 

2.  Modification  of  the  SCRR  model  as  necessary  in  order  to  apply  it 
at  the  TRACOM  level. 

3.  Review/modi  fixation  of  the  TPF  model  in  terms  of  the  following 
consult:  rations: 
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a  The  capabilities  of  a  DBMS  (e.  g.  ,  RAMIS)  to  supplant 
the  TPF  in  its  current  functions  (given  that  no  satisfactory  statistical 
parameters  can  be  developed). 

©  Possibilities  for  more  efficient  model  design  given  that 
the  week  by  week  simulation  is  equivalent,  in  effect,  to  successive 
transformations  of  course  parameters  using  vectors  of  proportionality 
factors  on  a  quarterly  basis. 

©  The  possibility  for  alternative  approaches  to  the  develop¬ 
ment  of  statistically  based  model  parameters. 

©  Modifications  as  necessary  to  accommodate  TRACOM  level 
application. 

4.  Design  and  fulfillment  of  field  tests  of  the  SCRR/TPF  at  TRALANT 
and  TRAPAC. 


5.  Presentation  of  the  ETE  model  to  the  core  of  ILS  course  developers 
in  the  EDTRACOM.  This  to  be  followed  by  field  testing  of  the  model 
in  several  cases  of  actual  course  design.  An  assessment  of  the 
benefits  of  the  model  should  be  made  following  these  tests  and  the 
model  should  be  further  developed,  revised,  or  put  on  the  shelf  for 
operational  use  as  ajrprqpriate. 

Recommendations  for  a  Future  Program  Resulting  From  DOTS 


1.  Review  training  management  problems  within  the  CNTT  command. 

A  logical  first  step  would  be  to  determine  if  and  how  the  Phase  I  analysis 
was  incorrect  or  incomplete  with  regard  to  its  description  of  the  CNTT 
functions.  However,  this  review  should  not  be  approached  from  the 
standpoint  of  how  the  presently  developed  Phase  II  models  can  be. 
made  1o  fit  CNTT  problems.  (If  this  is  a  possibility,  it  should  make 
itself  obvious  in  any  event  during  the  course  of  the  analysis.  ) 

Rather,  the  approach  should  be  to  determine  how  a  training  management 
suppend  program  (see  below)  could  best  serve  CNTT’s  needs. 


2.  At  the  conclusion  of  the  TRACOM  applicoiion/i'ield  test;,  a  program 
should  lie  ostablishe.i  devoted  to  the  use  and  development  of  manage¬ 
ment  science  techniques  to  support  training  management  throughout 
CNET.  The  agent  foi  this  program  should  reside  within  Lite  CNET 
community  but  have  close  ties  with  the  Is  £••  I )  sector.  This,  group  would 
serve  as  a  two-way  funnel  for  both  translating  CNET  R&.D  requirements 
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a  nr]  for  implementing  advancement;;  emanating  from  R&tD.  More 
importantly  the  group  would  have  the  capability  to  directly  address 
management  support  requirements  as  possible  within  the  state-of- 
the-art.  This  group  could  provide  an  invaluable  element  of  continuity 
and  consistency  in  the  solution  of  such  problems  within  CNET.  The 
DOTS  Phase  I  products,  as  corrected  or  amended,  should  provide  an 
excellent  foundation  for  this  program.  The  models  developed  thus  far 
would  be  the  start  of  a  "tool  kit"  of  models  or  other  techniques  that 
could  be  applied  to  problems  as  they  arise  within  the  training  communi 

3.  The  results  of  the  DOTS  project,  primarily  the  application  of 
data  base  management  technology,  should  be  integrated  more  closely 
with  the  work  of  tire  NETISA  organization  in  the.  development  of  the 
NITRAS  data  base. 

As  a  final  remark,  it  should  be  noted  that  while  the  objective 
of  the  DOTS  ADO  is  viewed  as  being  achieved,  the  magnitude  of 
expenditures  that  were  required  should  be  a  lesson  in  the  pitfalls  of 
setting  up  projects  which  arc  a  priori  tasked  with  pleasing  everybody. 
Ilit;  a  conviction  here  that,  at  least  for  the  present,  the  successful 
application  of  management  science  technology  is  much  more  likely 
through  many  small  scale  applications. 
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5.2  T est  and  Evaluation  Plan 


A.  Design.  The  design  focus  will  be  on  the  Naval  Education  and 
Training  Command  (CNET)  system  although  interfaces  with  the  total  Navy 
Education  and  Training  System  will  be  considered.  Experimental  models 

of  the  CNET  system  will  be  conceptualized,  formulated,  described,  developed 
and  computer-programmed.  The  models  will  possess  the  capability  of  accept¬ 
ing  different  parametric  values  (a)  of  variables  impinging  upon  the  CNET 
system  (e.g.,  mental  group  variations  in  input  populations  to  schools); 

(b)  of  variables  within  the  CNET  system  (e.g.,  available  school  seats  for 
a  particular  week,  month,  etc.);  and  (c)  of  variables  reflecting  state-of- 
the-art  condition  (e.g.,  reduction  in  training  time  through  the  use  of 
programmed  instruction).  Output  will  be  provided  in  the  form  of  cost  and 
effectiveness  measures, 

» 

B.  Test .  The  studies  will  be  designed  to  test  how  the  models  may 
be  used  for  effective  planning  and  decision-making  within  the  CNET  system. 
This  design  will  test  the  predictability  and  generalizability  of  the 
mathematical  models  to  the  entire  Naval  Education  and  Training  System. 

The  Fleet  Training  Center  (FLETRACEN),  Norfolk,  Virginia,  a 
sub-element  of  CNET,  and  Commander,  Training  Command,  U.  S.  Atlantic  Fleet 
(COMTRALANT) ,  has  been  tentatively  selected  as  the  test  bed  for  test  and 
evaluation.  The  FLETRACEN  was  selected  because  it:  (1)  has  an  extensive 
mix  of  schools;  (2)  is  subjected  to  unpredicted  requirements  for  new 
courses,  quotas,  etc.,  and  (3)  includes  all  functions  in  the  training 
development  cycle  between  course  design  and  course  implementation. 

The  test  design  will  include  experimental  variables  required  for 
test  and  evaluation  of  the  model  through  digital  simulation  and  will 
utilize  historical  data  where  available  for  assessing  the  required 
generalization  and  predictability  of  the  modeling.  In  this  manner,  the 
Naval  Education  and  Training  Command  will  be  provided  with  a  viable 
facility  for  ’effective  management  oontrol  over  the  Navy  Education  and 
Training  System,  and  a  consequentSgain  in  effectiveness,  quality,  and 
quantity  improvements  in  the  system.  The  on-site  test  will  be  conducted 
in  the  form  of  a  field  verification  by  an  independent  testing  activity. 

A  prdgram  for  continued  technical  improvements  in  the  models  and 
growth  to  simulation  will  be  provided,  as  well  as  recommendations  for 
incorporation-  of  educational  technology  features  into  the  selected  models. 
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5.2  Test  and  Evaluation  Plan  (Continued) 

C.  Evaluation.  The  final  determination  of  feasibility  will  be  in 
terms  of  technical,  operational,  and  financial  considerations. 

(1)  Technical  feasibility  evaluation  will  consider  the  state-of- 
the-art  in  mathematical  programming  models  applied  to  designing  training 
systems.  Consideration  will  be  given  to  areas  of  already  demonstrated 
applicability,  areas  requiring  additional  development,  and  areas  clearly 
needing  a  technological  breakthrough. 

(2)  Operational  feasibility  will  be  assessed  through  an  analysis 
of  the  ease  and  convenience  with  which  the  models  mav  be  utilized  by 
various  decision-making  echelons  within  the»CNET  organization.  Additionally, 
the  models  will  be  evaluated  as  to  their  ability  to  interface  with  other 
personnel  planning  and  forecasting  models  either  already  in  existence  or 
under  development. 

(3)  Financial  feasibility  will  be  determined  through  comparison 
between  the  personnel,  hardware,  and  support  costs  required  to  operate  the 
model  and  the  real  and  estimated  reductions  in  cost  anticipated  through 
use  of  the  models. 
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DEPARTMENT  OF  THE  NAVY 
TRAINING  ANALYSIS  AND  EVALUATION  GROUP 
ORLANDO,  FLORIDA  32813 

TAEG-.WHL 
8  April  197r< 


From:  Director,  Training  Analysis  .  nd  Evaluation  Group,  Orlando, 
Florida  32813 

To:  Commanding  Officer,  Naval  Personnel  Research  and  Development 

Center  (Attn:  F.  DiGialleonardo)  Bldg  200,  Washington,  D.  C. 

Sub  j  :  TDP  43-03,  Subproject  P01A,  Design  of  Training  Systems  (DOTS), 
test  and  evaluation  (T&E) ;  information  concerning 

Ref:  (a)  TAEC  ltr  TAEG:WHL  of  17  Jan  1975 

Enel:  (1)  Preliminary  User's  Test  Guide 

(2)  Test  and  Evaluation  Perspective  and  Goals 

1.  As  stated  in  reference  (a),  a  preliminary  User's  Test  Guide 
(enclosure  (1))  is  forwarded  for  possible  use  in  the  development  of  a 
test  and  evaluation  plan. 

2.  The  on-site  test  and  evaluation  of  the  DOTS  models  at  the  Fleet 
Training  Center,  Norfolk,  Virginia,  has  been  scheduled  for  23-27  June 
with  a  formal  training  course  for  T&E  members  16-20  June. 

3.  Representatives  from  the  Chief  of  Naval  Education  and  Training  and 
its  Functional  Commands  who  will  assist  NPRDC  in  the  performance  of  the 
formal  on-site  T&E  have  been  identified.  The  personnel  selected  are  as 
follows : 


CNET  Nr.  E.  Scheye 

Systems  Management,  N-336 
AV  922-3695 

COMPTRAPAC  LCDR  T.  L.  Ferrier 

Training  Analysis  and  ADP  Officer 
N-31 

AV  957-4219 

CQMTRALANT  Dr  .  Havey  Thorstad 

Education  Specialist,  FTC 
AV  690-4183 


B-l 


TAEG : WHL 
8  April  1975 


Sub j :  TDP  43-03,  Subproject  P01A,  Design  of  Training  Systems  (DOTS), 
test  and  evaluation  (T&E) ;  information  concerning 

CNETS  LCDR  R.  J.  Biersner 

Human  Factors  Analysis,  N-214 
AV  922-1392 

CNTECHTRA  CDR  J.  D.  Davis 

Assistant  for  Management,  015 
AV  966-5375 

Dr.  Norman  Kerr 
Training  Research,  0161 
AV  966-5593 


4.  Enclosure  (2)  provides  a  general  statement  of  the  overall  goals  of 
the  T&E  within  the  perspective  of  the  research  and  development  effort. 


5.  As  agreed  in  previous  discussions  between  TAEG  and  the  NPRDC  Washington 
Branch,  the  T&E  plan  should  be  completed  by  the  first  of  May  and  distribut¬ 
ed  to  the  T&E  representatives. 


A.  F.  SMODE 


Copy  to:  (w/enclosure  (2)) 

NPRDC  (J.  Silverman) 

CNET  (Mr.  E.  Scheye,  N-336) 

CNETS  (LCDR  R.  J.  Biersner,  N-214) 
COMTRAPAC  (LCDR  T.  Ferrier,  N-31) 
COMTRALANT  (Dr.  Harvey  Thorstad) 
CNTECHTRA  (CDR  J.  C.  Do  is.  Code  015) 
(Dr.  Norman  Kerc,  Code  0161) 


DEPARTMENT  OF  THE  NAVY 


NAVY  PERSONNEL  RESEARCH  AND  DEVELOPMENT  CENTER 
WASHINGTON  BRANCH  OFFICE 

BLDG  200  WASHINGTON  NAW  YARD 
WASHINGTON.  D.  C.  2O37U 

25  April  1975 

From:  DOTS  Test  and  Evaluation  Coordinator, 

NPRDC  Washington  Branch  Office 
To:  Distribution  List 

Subj:  DOTS  Test  and  Evaluation  Plan;  distribution  of 

Ref:  (a)  TAEG  ltr  TAEG:  whl  of  8  April  1975 

(b)  TDP  43-03,  Subproject  P01A,  Design  of  Training  Systems 
(DOTS),  April  1974. 

Enel:  (1)  Test  and  Evaluation  Plan  for  DOTS  Phase  IJ  Models 

1.  Reference  (a)  identified  representatives  from  the  Chief  of  Naval 
Education  and  Training  and  its  Functional  Commands  who  will  assist 
NPRDC  in  the  test  and  evaluation  of  models  produced  within  the  scope 
of  reference  (b).  These  representatives  will  comprise  a  User  Evalua¬ 
tion  Team  that  will  be  coordinated  by  the  undersigned  during  an  on-site 
T&E  to  be  conducted  at  the  Fleet  Training  Center,  Norfolk,  Va.  ,  23-27 
June. 

2.  Enclosure  (1)  provides  an  overview  of  the  T&E  in  general,  and 
the  specific  role  of  the  User  Evaluation  Team. 

3.  Comments  regarding  the  content  of  enclosure  (1)  are  solicited. 

n  -  J  A-  «  -4/- 

FRANK  DIGIA LLEONARDO 


Distribution: 

CNET  (Mr.  E.  Scheye,  N-336) 

CNETS  ( LCDR  R.  J.  Biersner,  N-214) 
COMTRAPAC  (LCDR  T.  Ferrier,  N-31) 
COMTRALAN7  (Dr.  Harvey  Thorstad) 
CNTECHTRA  (CDR  J.  C.  Davis,  Code  015) 

(Dr.  Norman  Kerr,  Code  0161) 

Copy  to: 

NPRDC  (Mr.  J.  Silverman) 

TAEG  (Mr.  H.  Okraski) 

(Mr.  V/.  Lindahl) 


Test  and  Evaluation  Plan  for  DOTS  Phase  II  Models 


Introduction 


Under  the  provisions  of  Technical  Development  Plan  (TDP) 

43-03.  01A,  April  1974  titled  Design  of  Training  Systems  (DOTS), 
the  Navy  Personnel  Research  and  Development  Center  (NPRDC)  has 
responsibility  for  conducting  a  test  and  evaluation  of  training  manage¬ 
ment  models  developed  in  Phase  II  of  DOTS.  This  plan  provides  an 
overview  of  the  manner  in  which  the  test  and  evaluation  will  be  conducted. 

Scope 

The  test  and  evaluation  (T&rE)  will  consider  technical,  operational 
and  financial  aspects  of  three  models  developed  under  DOTS:  (1)  the 
System  Capabiliti es  / R equi »-eme nts  and  Resources  Model  (SCRR);  (2) 
the  Educational  Technology  Evaluation  Model  (ETE):  and  (3)  the  Training 
Process  Flow  Model  (TPF).  In  addition,  capabilities  provided  for 
integrating  these  models  (where  applicable)  and  their  supporting  data 
base  will  be  similarly  evaluated. 

The  T&E  will  extend  to  other  DOTS  developments  in  addition  to  the 
above,  only  to  the  extent  that  they  impact  upon  the  basic  utility  and 
validity  ot  the  above  models.  The  T&cE  does  not  have  as  part  of  its 
objective  a  study  of  the  form  or  level  of  detail  required  for  DOD  approval 
of  Automated  Data  System  (ADS)  development.  However,  the  desirability 
and  feasibility  of  proceeding  to  such  a  shady  will  be  addressed. 

It  is  obvious  from  the  number  and  magnitude  of  possible  T&E  con¬ 
siderations  delineated  below'  (see  Approach)  that  a  full  assessment  of 
each  is  beyond  the  time  and  resources  available  to  the  T&E  study.  All 
of  the  considerations  will  be  addressed  to  some  extent;  those  most  in 
need  of  in  depth  analysis  will  be  pursued.  Other  areas  will  be  left  for 
further  study  as  may  be  desirable  based  upon  the  conclusions  and 
recommendations  of  the  T&tE. 

Re. sou  rces 

'Die  T&E  will  be  conducted  using  NPRDC  staff.  External  consulta¬ 
tions  will  be  made  with  respect  to  specific  technical  aspects  of  the 
models  as  required.  Representatives  of  prospective  users  of  the  models 
within  CNET  will  assist  in  the  T&E.  These  include  COMTR.APAC, 
COMTRALANT,  C NTT,  FTC  Norfolk,  CNETS  and  CNET.  Representativ 
of  the  Training  Analysis  and  Evaluation  Group  (TAEG),  and  the  DOTS 
monitors,  ?  nr?  of  thv  I H M  corp'jr?.tior.f  t.Hc  niodol  developers*  will  hie 
requested  to  provide  assistance  as  required.  Specific  areas  of  T&tE 
responsibility  are  indicated  below. 

Information  sources,  in  addition  to  the  above,  will  inc  lude  DOTS 
technical  reports,  IBM  conducted  validation,  test  procedures  and  results 
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activities  responsible  for  data  used  by  the  models,  data  on  operating 
cost  of  the  models,  and  other  prospective  users  that  may  be 
identified  during  the  course  of  the  T&E.  DOD  requirements  con¬ 
cerning  the  development  of  automated  data  systems  (e.  g.  ,  DOD 
Manual  4120.  17-M)  will  also  be  referenced. 

Approach 

Technical  Considerations.  Technical  considerations  as  used 
herein  refer  to  (1)  the  relationship  between  the  kinds  or  information 
produced  by  the  model  and  the  information  needed  by  management 
in  the  real  world,  (2)  the  manner  in  which  the  inform:.*  on  is 
developed  within  the  models,  and  (3)  the  data  upon  whi.  model  values 
are  based.  Relevant  subjects  include  the  following: 

a.  model  selection 

•  fit  of  model  to  problem 

•  complementary  aspects  of  the  models  to  each  other 

•  time  frame  for  model  operationalization 

•  potential  payoff  of  problem  area  addressed 

•  data  support  requirements 

•  level  of  decision  making  addressed 

•  acquisition,  analysis,  and  use  of  problem/process 
descriptive  data  to  develop  model  selection  and  design 
criteria 

b.  model  application 

•  selection  of  parameters 

•  simplifying  assumptions 

•  design  logic 

•  data  acces sability 

•  data  validity/accuracy 

•  manager-model  interaction 

c.  model  analysis  and  validation 


•  accuracy 

•  predictive  capability 

•  sensitivity 

•  method  of  validation 

•  representativeness  of  test  site  and  data  used  in  model 
development 


Operational  Considerations.  Operational  considerations  as  used 
herein  refer  to  factors  effecting  implementation,  use,  and  maintenance 
of  the  models  in  real  organizations.  Relevant  subjects  include  the 
following: 

a.  ease  and  practicality  of  use 

•  user  knowledge  requirements 

•  update  requirements 

•  response  time 

•  time  required  to  use  order  of  results  (i.  e.  ,  model  output) 

b.  institutionality  of  data  sources 

•  data  availability 

•  data  accessability 

•  data  management 

c.  organizational  implementation 

•  prospective  levels  of  application 

•  relationship  of  models  to  existing  policies  and  structures 
(i.e..  change  requirements) 

•  plan  for  implementation 

«  training  program(s)  for  facilitating  implementation  and 
reducing  resistance  to  change 

d.  magnitude  of  required  user  investment 

•  personnel  resources 

•  hardware 

•  maintenance 

•  other  fixed  and  variable  resource  requirements 

e.  method  of  operation 

•  stand  alone  (batch) 

•  interactive 

f.  supporting  docume ntation 


•  user  manuals 

•  program  documentation 


Financial  Considerations.  Financial  considerations  as  used 
herein  refer  to  the  cost  of  implementing,  supporting  and  maintaining 
the  developed  models.  The  objective  here  will  be  to  determine  if, 
and  to  what  extent,  it  is  financially  feasible  to  implement  the  models 
through  the  training  system.  Development  cost  of  the  models  to  date, 
while  a  consideration  of  the  T&E,  is  a  sunk  cost  with  respect  to 
further  implementation.  Relevant  subjects  include  the  following: 

a.  costs  of  implementation 

•  cost  of  developing  special  supporting  data  system,  if  any 

•  operating  cost  of  the  models 

-  projected  level  of  operation  (see  below) 

-  projected  extent  of  use  at  any  level 

-  ADP  support  alternatives  (e.  g.  ,  time  sharing, 
dedicated  system,  decentralized  system) 

•  cost  of  additional  model  development  or  integration 

•  cost  of  training  in  model  use  and  maintenance 

b.  range  of  potential  impact  and  levels  of  implementation 

•  minimum  level  of  implementation 

-  costs 

-  benefits 

•  maximum  level  of  implementation 

-  cost 

-  benefits 

•  implementation  alternatives 

c.  shelf  life  of  models 

•  model  structure 

•  model  factors  (e.g.,  ratios,  rates) 

d.  compatibility  of  models  to  existing  ADP  systems 

e.  computational  basis  for  assessing  benefit 
Di  s  cus  sion 

Most  of  the  considerations  delineated  above  have  been  addressed 
in  one  form  or  another  in  existing  and  soon  to  be  published  materials 
produced  by  TAEG/IBM.  This  information  will  be  closely  reviewed 


and  discussed  with  the  model  developers.  Independent  verification 
or  collection  of  additional  information  will  be  made  as  required. 

Many  of  these  points  will  be  explored  during  the  DOTS  model  training 
course  (16-20  June)  and  subsequent  on-site  user  test  and  evaluation 
(23-27  June). 

The  user  test  and  evaluation  will  take  advantage  of  the  presence 
of  representatives  from  prospective  CNET  users  to  focus  primarily 
upon  the  operational  considerations  of  the  models  as  outlined  above. 

A  model  evaluation  framework,  in  written  form,  will  be  provided  to 
members  of  the  user  test  and  evaluation  team  (9  June).  Users  will 
be  requested  to  utilize  this  framework  to  evaluate  the  models  during 
and  following  the  training  course.  The  user  evaluations  will  be 
ciscussed  on  an  individual  basis  for  the  purpose  of  obtaining  clarifica¬ 
tion  and  elaboration.  Group  discussions  will  be  held  among  the  user 
team  to  address  specific  issues.  IBM  or  TAEG  personnel  will  be 
called  upon  as  required  to  provide  further  explanation  concerning 
particular  aspects  of  the  models  to  the  team  members.  The  user 
team  will  collaborate  to  draw  up  a  summary  assessment  (draft)  at 
the  conclusion  of  these  discussions.  Additional  information  may  be 
required;  user  team  members  will  be  encouraged  to  discuss  the  models 
and  the  model  evaluation  framework  with  other  CNET  functionaries 
upon  return  to  their  respective  organizations.  NPRDC  will  coordinate 
the  composition  of  a  final  statement  of  user  evaluation  after  such 
additional  information  is  collected  from  the  members. 


Responsibilities 


As  stated  above,  the  user  evaluation  team  (i.  e.  ,  CNET  representa 
tives)  will  be  responsible  for  the  development  of  a  comprehensive 
statement  of  evaluation  focussing  on  operational  aspects  of  the  models. 
NPRDC  will  coordinate  this  i  ser  evaluation  and  take  primary 
responsibility  for  technical  and  financial  assessment,  (User  evaluation 
team  contributions  to  ther  latter  two  areas  will  be  welcomed  and, 


indeed,  may  be  specifically  requested  by  NPRDC.  )  TAEG  will  be 
responsible  for  the  provision  of  required  information  and  coordination 
with  IBM  and  CNET  functionaries.  NPRDC  will  be  responsible  for 
developing  a  final  report  detailing  the  results  arid  findings  of  the  T&tE 
effort. 

Schedule 


The  following  is  a  list  of  dates  relevant  to  the  T&E. 


1  May 


NPRDC  distribution  of  T&E  plan  to  TAEG 
and  CNET  participants 


31  May 
9  June 

lb-20  June 
23-27  June 
14  July 

31  August 
31  September 


-  Final  DOTS  test  plan  from  IBM/TAEG 

-  NPRDC  distribution  of  User  evaluation  framework 
to  CNET  participants 

-  DOTS  Training  Course,  FTC  NORFOLK 

-  User  test  and  evaluation;  draft  evaluation  statements 

-  Final  inputs  from  CNET  test  and  evaluation 
participants  to  NPRDC  coordinator 

-  Completion  of  T&E  and  report  of  findings 
DOTS  final  document  package 
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DEPARTMENT  OF  THE  NAVY 


NAVY  PERSONNEL  RESEARCH  AND  DEVELOPMENT  CENTER 
WASHINGTON  BRANCH  OFFICE 

BLOG  200  WASHINGTON  NAVY  YARD 
WASHINGTON,  D.  C.  20374 


NPRDC:FD:gbg 
3  June  1975 


From:  DOTS  Test  and  Evaluation  Coordinator,  NPRDC 
Washington  Branch  Office 
To:  Distribution  List 

Subj:  DOTS  Test  and  Evaluation,  23-27  June  1975,  FTC 

Norfolk,  Va. 

Ref:  (a)  NPRDCjWBO  ltr  to  distribution  list,  25  April  1975, 

Subj:  Distribution  of  T &E  Plan 

Enel:  (1)  Schedule  and  Outline  for  User  T &E  of  DOTS  Models 

(2)  DOTS  Expanded  Milestone  Chart  and  Assignments  for 
IBM,  T&ET  and  FLETRACEN,  23  April  1975 

1.  Reference  (a)  forwarded  an  overview  of  the  T &E  as  it  will  be 
approached  by  NPRDC.  As  stated  in  that  plan,  it  is  essential  that  the 
subject  T&E  result  in  a  clear  assessment  of  the  operational  feasibility 
of  the  DOTS  models.  Towards  that  end,  enclosure  (1)  is  forwarded  for 
info r mation,  review  and  comment.  The  schedule  may  be  adjusted  on 
the  basis  of  comments  received  from  the  team  members  (preferably 
via  phone  before  16  June). 

2.  It  is  understood  that  an  assessment  of  operational  feasibility  cannot 
be  made  totally  independent  of  technical  and  financial  feasibility  con¬ 
siderations.  It  is  expected  that  information  concerning  the  latter  areas 
will  develop  during  the  course  of  subject  User  T&E  at  Norlolk.  However, 
the  main  responsibility  for  these  two  areas  will  be  borne  by  NPRDC 
extrinsic  to  the  subject  User  T&E.  Informal  support  from  the  User 
T6E  team  members  is  likely  to  be  sought  during  this  endeavor.  But, 
due  to  the  time  constraint  on  full-time  team  member  participation,  it 

is  prudent  that  the  subject  User  T&E  focus  »B  sharply  as  poss,ble 
upon  operational  feasibility  considerations. 

3.  It  is  expected  that  each  team  member  will  be  able  to  make  substantial 
progress  towards  an  evaluation  of  the  models  as  a  by-product  of  the 
training  session  in  DOTS  model  usage  (16-20  June). 
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4.  Enclosure-  (1)  is  a  fairly  detailed  schedule  and  outline  for  the  T&E 
week.  The  assessment  of  the  models  and  data  base  will  be  approached 
through  five  major  discussion  areas.  There  will  be  a  discussion  leader 
and  recorder  for  each  area.  All  five  areas  will  be  covered  within  the 
first  three  days  of  the  week,  leaving  the  last  day  and  a  half  for  the  com¬ 
position  of  a  draft  T&E. 

The  responsibilities  of  each  discussion  leader  will  be  as  follows: 

a.  Develop  essential  points  .  ;•  discussion  (enclosure  (1)  contains 

a  good  start  towards  these). 

b.  Pace  the  discussion  aid  observe  time  constraints. 

c.  Follow-up  questions  which  remain  unanswered  after  the  dis¬ 
cussion  session  (time  is  provided  in  the  schedule  for  this  activity). 

d.  Review  written  summary  of  the  discussion  in  his  area,  as  well 
as  notes  made  by  the  team  members  during  the  course  of  the  discussion. 

e.  At  a  later  time  (see  schedule)  take  the  lead  in  developing  a 
draft  section  for  his  discussion  area. 

The  responsibilities  of  the  recorder  will  be  as  follows: 

a.  Make  a  written  record  of  the  essential  remarks  made  during 
the  discussion,  as  well  as  the  summarization  and  any  subsequent 
modifications.  (This  written  record  need  not  be  in  smooth  or  narrative 
form,  nor  is  it  essential  to  stipulate  who  said  what.  The  important 
thing  will  be  to  get  the  remarks  in  writing  so  that  they  can  be  later 
reviewed  by  the  team  and  ultimately  developed  into  a  draft  T&E  state¬ 
ment  by  the  discussion  leader.  ) 

b.  See  that  the  notes  get  to  the  typist  as  soon  as  possible  and  that 
they  are  distributed  to  the  team  members  by  the  afternoon  of  the 
following  day. 

A  verbal  summarization  of  each  discussion  will  be  made  by  the  TbE 
coordinator  at  its  conclusion.  Following  this,  omissions,  corrections 
and  modifications  as  noted  by  the  team  will  be  invited  and  recorded. 

Each  team  member  will  be  responsible  for  making  notes  on  dis¬ 
cussion  points  that  he  sees  as  particularly  important.  These  notes  should 
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In-  convoyed  to  the  recorder  or  discussion  leader  as  appropriate.  Each 
member  will  also  be  responsible  for  making  comments  as  necessary  on 
the  recorded  summaries  of  each  session  and  providing  these  to  the 
appropriate  discussion  leader  for  inclusion  in  the  draft. 

All  team  members  will  participate  in  the  formulation  of  a  draft  T&E 
assessment  during  the  last  two  days  of  the  week.  The  T&E  coordinator 
will  be  responsible  for  transforming  the  draft  into  a  smooth  version 
subsequent  to  the  T&.E  week. 

It  is  expected  that  the  various  DOTS  models  will  be  available  for 
further  "hands  on”  experience  during  the  T&E  week.  Team  members 
are  encouraged  to  bring  to  the  T&E,  representative  test  problems  that 
might  be  tried -out. 

5.  Enclosure  (2)  contains  additional  and  summary  information  on  the 
DOTS  program.  The  "Test  aiid  Evaluation  Goals"  stated  therein  do  not 
constrain  or  modify  the  T&tE  plan  as  outlined  in  reference  (a)  or 
partially  detailed  in  enclosure  (1). 

6.  I  am  looking  forward  to  working  with  each  of  you.  A  valid  assessment 
of  the  models  developed  under  DOTS  is  a  formidable,  though  necessary 
task.  I  am  pleased  that  the  T&E  will  be  able  to  benefit  from  your 
expertise  in  the  area  of  training  management  and  from  your  insight  to 

the  requirements  of  the  CNET  community. 


Sincerely, 


FRANK  DIGIA  ELEONARDO 
Area  Code  (202)  433-4760 
Autovon  288-4760 


Distribution  List 

CNET  (Mr.  E.  Scheyc,  N-336) 

CNETS  ( LCDR  R.  J.  Biersner,  N-214) 

COM  TR  A  PAG  (LCDR  T,  Ferrier,  N-31) 
COMTRALANT  (Dr.  Harvey  Thorstad) 
CNTECHTRA  (CDR  J.  D.  Davis,  Code  015) 

(Mr.  David  Thomas,  Code  0161) 


Copy  to: 

NPRDC  (Mr.  J.  Silverman) 
(Dr.  R.  Sorenson) 
(Dr.  J.  Regan) 

(Mr.  E.  Hooprich) 
T AEG  (Mr.  H.  Okra  ski) 
(Mr.  W.  Lindahl) 
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SCHEDULE  AND  OUTLINE  FOR  USER  TixE  OF  DOTS  MODELS 

16-20  JUNE  1975 
FTC,  NORFOLK,  VIRGINIA 


23  June  1975  -  Monday 

8:00  Introductory  Remarks 

8:15  individual  team  members'  discussion  of 
particular  interests  and  objectives  in 
the  T&E. 

9:00  Presentation  and  Discussion  of  Week's 
Schedule;  adjustments  as  necessary. 

9:30  Assignment  of  Duties 

Discussion  Leader 
Recorder 

Summarizat ion /Feedback 

Additions,  exceptions,  modifications 

10  : 00  Break 

10:15  First  Discussion  Area:  Potential  Model 
Contributions 


Discussion  Leader:  DiGial leonar do 
Recorder  :  Thorstad  [changed  to  Brooks] 
Summarization:  DiGi a  1 leonardo 

A.  At  the  Activity  Level 

•  genera]  nature  of  contributions 

•  specific  areas  of  contribution 
(e.  g.,  which  training  functions, 
or gamzat  ion) 

•  source  of  contribution  (i.  e.,  which 

mode  1 ) 

•  ranking  of  contributions  by  area/ 
ro  de  1  u  .  g  .  , 


Enclosure 


(1) 


f 


0  none 

1  low 

2  moderate 

3  substantial 

4  very  high 

•  importance  of  application  area  co 
CNET/CNO 

(e.g.,  same  ranking  as  above) 

B.  At  other  levels  (e.g.,  sub-activity 
level,  larger  commands,  CNET  level). 

•  (similar  considerations  as  for  A 
above) . 

C.  Other  discussion  as  necessary. 

3:30  D.  Summarization 

4:00  E.  Omissions,  Corrections,  Modifications 
Adjourn:  4:30 
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24  June  1975  -  Tuesday 


8:00  Second  Discussion  Area:  Ease  and  Practi¬ 
cality  of  Use 

Discussion  Leader:  Biersner 

Recorder :  Thomas 

Summar ization :  DiGialleonardo 

A .  User  Knowledge  Requirements 

•  Knowledge  about  the  problem 

What  level  of  decision  making 
expertise  is  needed  to  formulate 
the  problem? 

To  obtain  model  results? 

To  utilize  the  results  of  the  model 
run? 

•  How  much  time  is  necessary  and  what 
difficult  ^s  are  involved  in  obtain¬ 
ing  abo\  knowledge  requirements? 

B  .  Update  P  acruirements 

•  What  update  requirements  exist  for 

the  models  and  their  data  base?  (e.g., 
student/instructor  ratios,  student 
profile/course  performance  relationship, 
etc . ) 

Iiow  extensive  are  these  update 
requirements? 

Can  they  be  easily  supported  within 
current  training  activity  operations? 

•  What  level  of  expertise  is  needed  to 
make  the  updates? 

•  Hew  frequently  would  they  probably  need 
to  occur? 
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C .  Response  Time 

•  How  much  time  is  required  to  prepare 
a  typical  model  run? 

•  How  much  time  is  required  to  make  a 
typical  model  run? 

•  How  much  time  is  required  to  analyze 
and  apply  the  results? 

D .  Output  Form 

•  Is  the  form  and  level  of  detail  of  the 
model  output  amenable  to  direct  usage 
or  must  the  user  make  significant 
transformations? 

•  How  extensive  are  the  required  trans¬ 
formations? 

•  In  the  typical  problem  how  much  of 
the  solution  comes  from  the  model  and 
how  much  from  other  sources  (e.g., 
discretion  of  the  decision  maker)? 

E.  Other  discussion  as  necessary. 

F .  Summarization 

G.  Omissions,  Corrections,  Modification' 
(11:30  -  12:30  Lunch) 

12:30  Distribution  of  previous  day's  summary  for 

review. 

12:45  Third  Discussion  Lrea :  Data  Requirements 


Discussion  Leader:  Scheye 
Recorder:  Ferrier 

Summarization:  DiCialleonardo 


Nature  of  the  Data  Utilized  by  Models 


•  Are  the  data  valid?  (i.e.,  do  they 
measure  or  otherwise  represent  the 
right  phenomena  given  their  use  in 
the  model?) 

•  Are  the  data  reliable  (i.e.,  are 
they  consistently  reported  from  one 
time  period  to  another?  is  the 
method  of  data  reporting  standardized?) 

•  Are  the  data  accurate?  (i.e.,  given 
what  they  are  to  represent,  is  the 
representation  usually  correct?) 

Data  Availability 

•  Are  the  data  required  presently  in 
existence? 

•  If  not,  what  would  it  take  (in  time 
and  money)  to  institute  the  data 
source  and  reporting  mechanism? 


Data  Accessabilit 


E 


Future  Data  Problems 


•  Which  of  the  above  data  considerations 
would  become  critical  as  the  models 
are  spread  throughout  CNET  and  as 
attempts  to  aggregate  data  and  model 
results  are  made? 

F.  Other  discussion  as  necessary. 

G.  Summarization. 

H.  Omissions,  Corrections,  Modifications. 
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25  June  1975  -  Wednesday 

8:00  Fourth  Discussion  Area:  Organizational 
Implementation 

Discussion  Leader:  Davis 
Recorder :  Scheye 

Summarization:  DiGialleonardo 

A.  Prospective  Levels  of  Application 

•  Within  activities. 

•  Beyond  activities. 

•  Do  the  models  support  current  decisions 
or  decisions  which  should  be  made  but 
currently  are  not? 

•  In  the  latter  case,  what  roadblocks 
exist  to  creating  such  decision-making 
processes? 

B.  Relationship  of  Models  to  Existing  Policies 
and  Structures 

•  Do  the  models  support  or  seek  to  change 
current  policies  and  structures? 

Are  various  CNET  activities  likely  to 
be  receptive  in  either  rase? 

What  are  the  incentive's  for  pro¬ 
spective  model  users  to  become  actual 
model  users? 

Have  possible  bases  of  resistance  been 
considered  in  the  model  development 
program? 

Has  a  specific  plan  for  implementation 
been  formulated  or  has  an  implementa¬ 
tion  strategy  been  worked  out? 


How  well  do  the  models  fit  in  with 
future  plans  for  the  CNET  organization? 


C.  Implementation  Support 


•  What  is  the  quality  and  coverage  of 
the  training  programs  that  have  been 
developed? 

•  What  is  the  quality  and  coverage  of 
the  user's  manuals  and  other  documenta¬ 
tion  that  has  been  developed? 

•  What  CNET  organization (s)  would  sponsor 
and/or  support  the  implementation  of 
the  models? 

D.  Other  discussion  as  necessary. 

E.  Summarization. 

F.  Omissions,  Corrections,  Modifications. 


(11:30  -  12:30  Lunch) 

12:30  Distribution  of  Previous  day’s  summaries  for 
review . 

12:45  Fifth  Discussion  Area:  User  investment 
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Discussion  Leader:  Ferrier 

Recorder:  Biersner 

Sununar izat ion  :  DiGialleonardo 

.  Personnel  Resources 

0  Would  specially  skilled  people  have 
to  be  acquired  or  could  present  staff 
learn  to  operate  models  or  use  results? 

•  How  much  additional  staff  time  would 
be  required  to  operate,  maintain  and 
utilize  the  models  and  their  data  base? 
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Hardware 

•  What  hardware  acquisitions  would  typi¬ 
cally  have  to  be  made? 


■*.  •*.  < .  .  .*  -•  .  _  .* 
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What  is  their  approximate  cost? 


•  What  would  be  approximate  time  sharing 
or  other  ADP  charges  for  main  frame 
usage? 

•  Given  findings  in  First  and  Fourth 
discussion  areas,  what  might  be  a 
"ballpark"  cost  per  incidence  of 
usage  ratio? 

C.  Method  of  Operation 

•  Stand  alone? 

•  Time  sharing? 

•  Other? 

D.  Other  discussion  as  necessary. 

E.  Summarization. 

F.  Omissions,  Corrections,  Modifications. 


i 
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26  June  1975  -  Thursday 


8  :00 


Each  discussion  leader  will  prepare  a  draft 
of  the  T&E  Team's  findings  with  respect  to  his 
discussion  area.  Dr.  Thorstad  will  assist  CDR 
Davis  with  the  draft  on  the  fourth  area  (Organi¬ 
zational  implementation) .  Mr.  Thomas  will  assist 
Mr.  Scheye  with  the  draft  on  the  third  area  (data 
requirements) .  Unanswered  questions  arising  out 
of  the  previous  days  discussion  sessions  or  the 
composition  of  the  drafts  will  be  followed-up 
through  calls  to  the  appropriate  TAEG  or  IBM  repre¬ 
sentatives  (coordinated  through  Mr.  Lindahl)  or  to 
other  sources  as  required. 

Final  summarizat ions  from  Wednesday  should  be 
distributed  by  noon.  Comments  on  each  summary 
should  be  communicated  to  the  appropriate  discussion 
leader  for  consideration  in  the  draft. 

Typing  of  drafts  should  begin  as  soon  as 
possible. 
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27  June  1975 


Friday 


9:30  Begin  review  of  draft  segments. 

12:00  Completion  of  draft  review;  adjournment 

Completed  working  draft  (smooth,  final  version 
will  be  developed  by  the  T&E  coordinator  at  a  later 
date.  Members  will  be  encouraged  to  take  copies  of 
the  draft  back  to  their  respective  commands  for  re¬ 
view  and  comments  that  might  be  later  included  in  the 
final  version.) 


APPENDIX  F. 


TEST  PROBLEMS  AND  CRITERIA  FOR  EVALUATION 
OF  POTENTIAL  MODEL  CONTRIBUTIONS 
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The  following  pages  contain  a  listing  of  possible  problem  areas 
towards  which  tho  models  developed  in  the  DOTS  project  might  be 
effectively  directed.  Most  of  these  problem  areas  were  identified 
through  discussions  with  training  managers  at  FTC,  Norfolk.  Also  ; 
included  arc  two  lists  of  questions  relating  to  these  problem  areas. 

In  providing  the  information  requested,  it  is  suggested  that  you  ; 
first  scan  the  outline  of  possible  problem  areas,  circling  each  one 
that  has  relevance  to  your  particular  CNET  organization/function. 

Any  additional  problem  areas  that  seem  to  be  relevant  but  not 
included  in  the  outline  should  also  be  noted  for  subsequent  evaluation. 
Next,  proceed  to  the  criteria  of  Part  One  and  respond  to  each  with 
respect  to  each  problem  area  that  you  have  identified  as  relevant. 

(A  separate  criteria  list  should  be  filled  out  for  each  relevant  problem 
area.)  Finally,  answer  the  questions  in  Part  Two.  (These  need  be 
answered  only  once.  ) 

Please  provide  your  name,  position  and  name  of  your  organization/ 
function  below. 
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TEST  PROBLEM  AREAS 

I.  Instructor  Personnel  Recap  Report 

A.  ”  To  provide  the  trainin']  program  coordinators  with  six 
naonch  p’icturo  of  a  school's  instructor  status  In  order  for  him  to 
detail  incoming  Instructors  where  most  needed. 

B.  Needed 

1.  Instructor  allowance  list 

2.  Instructors  assigned  versus  allowance  -  personnel  filling  the 
allowed  billets  combined  with  a  listing  of  assignments  not 
cover'd  by  a  billet. 

3.  Totals  recapitulation:  Allowed  On  Board 
Rata  for  rate 

NEC  for  NEC 
Total  for  total. 

4.  Specific  course  information: 

a.  Instructors  required 

b.  Instructors  qualified 

c.  Instructors  rotation  dates. 

II.  Effect  on  number  of  instructors  required  If: 

A.  Student  throughput  is  increased 

B.  Student  throughput  is  decreased 

C.  Student  decrease  required  to  permit  ‘X1  reduction  In  number  of 
instructors. 

III.  Contact  Hours 

A.  Total  contact  hours  at  FLETRACEN 

B.  How  many  instructors  are  required  for  an  average  of: 

30  contact  hours 
25  contact  hours 
20  contact  hours 
15  contact  hours 
10  contact  hours 

C.  Same  questions  as  * B *  broken  out  by  each  school. 

IV.  Lowering  attrition  rate: 

A.  If  a  student  is  a  volunteer  for  the  course,  all  other  factors  being 
equal,  how  much  more  likely  is  he  to  pass  than  a  non- volunteer? 

B.  What  effect  docs  studont/instructor  ratio  have  on  attrition  rate? 

1.  If  GCT/ARl  lowered  but  desire  to  maintain  a  low  attrition  rate 
what  student/instructor  ratio  would  accomplish  this? 

2.  What  student/instructor  ratio  is  required  to  reduce  attrition  rate? 
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(Continued) 

C.  What  attrition  effects  can  be  expected  from  course  lengthening  or 
shortening: 

1.  All  other  things  being  equal. 

2.  Decrease  in  instructors. 

.3.  Decrease  In  GCT/ARI  entrance  requirements. 

Cross-Training 

A.  Given  the  courses  which  are  amenable  to  cross-training  plus  the 
qualificrti'"*?  required  to  teach  that  course,  determine: 

1.  Which  schools  contain  the  instructors  meeting  eligibility 
requirements. 

2.  Average  contact  hours  of  that  school. 

3.  Number  of  instructors  available  for  cross- training. 

Note:  A  factor  will  be  the  qualification  time  in  the  new 

course  and  whether  or  not  an  instructor  can  be  re¬ 
leased  for  that  period  of  time. 

4..  Once  qualified  what  will  be  criteria  for  maintaining  qualifications? 
Teach  'X'  times  per  year? 

B.  An  accounting  system  will  be  required  in  order  to  readily  identify  to 
comr.and^  cross- trained  instructors. 

Personnel  Cuts 

Given  a  requirement  for  an  'X'  percent  personnel  cut  with:' 

A.  No  corresponding  reduction  in  students 

B.  Corresponding  reduction  in  students 
For  each  case  determine: 

1.  Conti ciering  course  priority  system  which  courses  should  be  cut? 

2.  Based  on  course  utilization  which  courses  should  be  cut? 

Convening  Frequency 

If  course  convening  frequency  is  reduced  because  it  is  experiencing 
a  low  utilization  rate: 

A.  Will  the  customers  use  the  course  less  because  the  convening  dates 
are  not  convenient? 

a.  Are  the  requirements  for  graduates  being  fulfilled  and  are  we  doing 
this  only  because  of  the  convenience  of  convening  dates  factor? 

C.  Should  class  capacity  be  reduced? 

If  this  is  done  will  the  number  of  Instructors  required  be 
reduced? 

2.  Is  the  no-show  rate  so  high  that  reducing  class  capacity  produces 
the  same  effect  as  reducing  convening  frequency? 

3.  Is  the  attrition  rate  so  high  that  this  also  creates  the  same 
effect? 


VIII.  Faciliti.es  Ut i  1  iza t  ion/Capac it y 

If  Facilities  are  increased/decreased  by  "x", 

A.  How  many  more/less  courses  can  be  offered? 

B.  How  many  more/iess  instructors  are  needed  at 
full  utilization? 

C.  How  many  more/less  students  can  be  trained? 

IX.  Resource  Interrelationships 

A.  Given  a  certain  change  in  student  throughout 
requirements , 

1.  What  combinations  of  instructor  resources, 
facility  resources,  and  c-au r  ccnvcn lag 
schedule  will  permit  the  requirement  to  be 
met  ? 

2.  What  is  the  best  combination  the  above  factors 
in  terms  of 

a.  most  efficient  use  of  instructors? 

b.  most  efficient  use  of  facilities? 

c.  most  convenient  course  convening  schedule 
for  customers? 

B.  Given  a  change  in  courses  demanded  (required),  what 
is  the  effect  on 

1.  instructors  required  (quantity  and  quality)? 

2.  facilities  required/utilized? 

3.  student  throughput  capacity? 

4.  number  of  convenings  that  can  be  scheduled? 
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TEST  PROBLEM  CRITERIA 


PART  ONE 

For  each  test  problem  area  that  can  be  identified  from 
the  preceding  outline  (e.  g.,  IV.  B.  2.)  or  for  any  other 
pertinent  problem  area  faced  by  your  organization,  please 
respond  (vO  to  the  following  criteria.  (Multiple  criteria 
lists  are  provided) . 

CRITERIA  LIST 

Problem  area  or  decision:  (Use  code  such  as  IV.  B.  2  from 
outline  or  describe  in  narrative  form.) 


How  frequently  is  this  problem  encountered? 

a.  yearly 

b.  semi  annually 

c.  monthly 

d.  weekly 

e.  daily 

f.  never  (If  never,  ignor  following  questions  and 

proceed  to  next  problem  area.) 

g.  other  state:  _ 

V.Ti a t  is  the  magnitude  of  the  problem  as  it  currently 
exists? 


little  or 


minor 


substantial 


major 


critical 


How  adequate  are  present  methods  of  dealing  with 
this  problem? 


adequate  all  of  the  time 
adequate  most  of  the  time 
inadequate  most  the  time 
never  adequate 


4.  How  likely  is  the  problem  to  be  significant  in  the 
near  future  (e.  g.,  up  to  1980)? 


Virtually  Not 

Impossible  Likely 


Like ly 
as  Not 


Likely 


Virtually 

Certain 


5.  How  sizeable  is  the  opportunity  for  improving  on 
present  decisions  or  practices  currently  used  in 
this  problem  area? 


Minimal 


Small 


Moderate 


Substantial  Great 


Why? 


6.  What  order  of  savings  (annual)  might  be  realized 
through  effective  solution  of  the  problem  on  a 
regular  basis?  (For  the  subject  organization  only.) 

a.  million(s)  ($) 

fc.  hundred (s)  of  thousands 

c.  ten(s)  of  thousands 

d.  hundred (s)  of  dollars 

e.  less  or  none 

Provide  rough  statement  of  rationale:  _ 


What  order  of  non  monetary  benefits  could  be  derived 
from  effective  solution  of  the  problem? 


Minimal 


Small 


Moderate 


Substantial  Great 


8.  How  much  of  a  change  in  the  organization's  structure 
or  decision  making  process  would  be  required  for 
effective  solution  of  the  problem? 


Minimal 


Small 


Moderate 


Substantial  Great 


PART  TWO 


The  models  developed  deal  with  certain  major  factors  of  the 
training  environment.  Rased  upon  your  experiences  and  information, 
assess  the  variability  (  i/  )  of  these  factors  as  it  occurs  in  the  real 
training  environment  within  the  normal  planning  horizon. 

1.  Student  throughput  (training  demand) 


a.  qualitative  variability  (i.  e.  ,  changes  m  Kindt-  or  types  o: 
students ) 


1 _ _ 1 _  _  1 

relatively 

mode  rately 

widely 

fixed 

va  riable 

va  ri able 

b. 

quantitative  variability  (i.e.  , 

changes  in  numbers  of 

1 _  _ 

students) 

1 

1 

relatively 

moderately 

wi  d  e 1 y 

fixed 

variable 

variable 

o  7 

tructors 

, 

a. 

qualitative  variability  (i.e.  ,  changes  in  kinds  or  types  of 

instructors ) 

{ _ _ 

i 

i 

relativel  y 

moderately 

widely 

fixed 

variable 

variable 

b. 

quantitative 

variability  (i.  e.  ,  changes 

in  numbers  of 

1 _ 

instructors) 

.  .  i 

i 

relatively 

mode  rately 

widely 

fixed 

va  riable 

variable 

3.  Faciliti  es 

a. 

Qualitative 

facilities) 

variability  (i.e.,  changes 

in  kinds  or  types  of 

i 

-i 

i 

Relati  vc  1  v 

moderately 

wi  vi  cl  y 

fixed 

va  ri  able 

variable 
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relatively 

fixed 


moderately 

variable 


v*  i  d  <  1  y 
va  riablc 


4.  Courses 

a.  qualitative  variability  (i.  e.  .  changes  i  r.  l.l  i.f 

t _ 1 _ _ _ i 

relatively  moderately  widely 

fixed  variable  variable 

b.  quantitative  variability  (i.e.  ,  changes  in  numb.-rs  of 
courses,  corvonings) 


l _ 

relatively 

fixed 


_ I _ 

moderately 

variable 


_ i 

widely 

variable 
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DEPARTMENT  OF  THE  NAVY 

CHIEF  OF  NAVAL  EDUCATION  AND  TRAINING  SUPPORT 

PENSACOLA,  FLORIDA  32COQ 


Code  N-214 
3900 

8  AUG  1975 

Chief  of  Naval  Education  and  Training  Support 
Director,  Naval  Personnel  Research  and  Development 
Center,  Washington  Branch  Office,  Bldg  200,  Washington 
Navy  Yard,  Washington,  D.C.  20374 

Design  of  Training  Systems  (DOTS) ;  comments  on 
evaluation  of 

(a)  FONECON  between  CNETS  Code  N-214  and  Mr.  Frank 

DiGialleonardo  of  NAVPERSRANDCEN,  Washington  Branch 
Office  on  28  Jul  1975 

(1)  Draft  copy  of  DOTS  Test  and  Evaluation  User  Evalua¬ 
tion  dated  Jun  1975 

1.  Enclosure  (1)  has  been  reviewed  for  editorial  correction 
as  requested  in  reference  (a) .  It  is  recommended  tnat  the 
final  draft  contain  more  data  from  user  questionnaires  which 
were  administered  during  the  DOTS  evaluation  held  at  COMTRALANT 
in  July,  especially  documentation  on  bases  for  cost-savings 
estimates. 

2.  The  CNETS  would  like  to  take  this  opportunity  to  recom¬ 
mend  a  more  extensive  field  evaluation  of  the  DOTS  models, 
preferably  at  several  training  commands  which  have  student 
and  instructor  resources  which  could  best  be  served  by  the 
DOTS  models.  It  is  recommended  that  this  evaluation  be 
performed  by  a  Navy  activity,  and  that  it  carefully  document 
upper  echelon  requests  for  training  management  information 
which  could  be  provided  by  the  DOTS  models,  times  and  costs 
involved  in  personnel  training  (systems  analysts  and  opera¬ 
tors)  ,  effectiveness  of  training,  times  and  costs  required 

to  operate  the  hardware  systems,  arid  the  frequency  with  which 
each  model  was  used.  Any  cost-saving  which  results  from  use 
of  the  DOTS  models  should  be  documented,  as  well  as  critical 
decisions  that  were  expedited  through  use  of  these  models. 

It  is  further  recommended  that  an  effort  be  made  to  interface 
these  models  with  available  input/output  hardware  at  these 
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Code  N-214 
3900 


evaluation  sites,  and  that  alternatives  to  the  existing  data 
base  management  system  also  be  explored. 

By  direction  . 

Copy  to  w/o  end: 

CNET  (N-56) 

TAEG 

r  i 


-j 


ir  Al 


tw  -  V  w 


CHIEF  OF  NAVAL  EDUCATION  AND  TRAINING 
NAVAL  AIR  STATION 
PENSACOLA,  FLORIDA  32906 

Code  N-562 
5  August  1975 


MEMORANDUM 

From:  Code  N-562,  Chief  of  Naval  Education  and  Training 

To:  Mr.  Frank  Digialleonarc’o,  NPRDC  Washington  Branch  Office, 

Bldg.  200,  Washington  Navy  Yard,  Washington,  D.C.  20374 

Sub j :  Review  of  DOTS  Test  and  Evaluation,  User  Draft  Report 

Ref:  (a)  DOTS  T&E  meeting  of  23-27  June  1975 

Enel:  (1)  Annotated  User  Draft 

(2)  Recommended  references 

1.  In  accordance  with  the  general  agreements (s)  reached  in  reference  (a) 
enclosure  (1)  is  forwarded.  No  substantive  changes  were  made.  A  final 
draft  which  incorporates  the  input  provided  by  other  T&E  respondents  is 
requested,  prior  to  final  approval.  Enclosure  (2)  Is  provided  as  a  ready 
reference  for  determining  compliance  with  specific  requirements  as 
delineated  by  higher  authority. 

2.  Continued  communication  on  this  project  is  desired.  Please  feel 
free  to  address  any  question  germaine  to  DOTS  to  the  undersigned. 


COMMANDER  TRAINING  COMMAND 
UNITED  STATES  PACIFIC  FLEET 

SAN  DIEGO,  CALIFORNIA  92147 


FF8-1  /  31  :rtb 

sT/W 

16  SEP  1975 


To:  Navy  Personnel  Research  and  Development  Center,  Washington 

Branch  Office  (Attn:  Mr.  Prank  DiGialleonardo) 


Sub  j :  Design  of  Training  Systems  (DOTS)  (TDP  43-03,  Sub-Project  P01A) 


Enel:  (1)  COMTRAPAC  Position  Concerning  DOTS 


1.  The  DOTS  Project  is  presently  undergoing  test  and  evaluation,  a 
feature  of  which  is  an  assessment  of  potential  user  requirements  for 
the  product  of  the  project. 

2.  Enclosure  (1)  is  provided  as  a  statement  of  position  vis-a-vis 
DOTS. 


3.  Commander  Training  Command,  U.  S.  Pacific  Fleet  has  been  experimenting 
with  access  to  the  DOTS  models,  and  although  certain  equipment  interface 
problems  remain  unresolved,  access  appears  feasible  without'"*  significant 
additional  investment  in  either  hardware  or  software . 


4.  Should  further  information  of  use  in  the  T&E  be  developed,  it  will 
be  forwarded  to  the  Navy  Personnel  Research  and  Development  Center, 
Washington  Branch  Office. 


Copy  to: 

TAEG ,  Orlando 
CNET 


3 .  E.  EVAJfS 
Chief  of  Staff 
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ORIGINATOR:  LCDR  T.  L.  FERRIER 
CODE ;  N31 


COMTRAPAC  SDIEGO 
UNCLASSIFIED 

A.  SUBJECT/PRORLKM :  DESIGN  OF  TRAINING  SYSTEMS  (DOTS)  (TDP  43-03 
SUB  PROJECT  r01A~) 

B.  FACTS/DISCUSSION :  THE  DOTS  PROJECT  RECOGNIZES  THE  NEED  FOR  INTRODUCING 
OPERATIONS  RESEARCH  AND  MANAGEMENT  SCIENCE  TECHNIQUES  INTO  THE  NAVEDTRACOM . 
THESE  TECIS ION-AIDING  MODELS  ARE  INTENDED  TO: 

1.  INCREASE  EFFICIENCY  OF  TRAINING  RESOURCE  UTILIZATION. 

2.  OPTIMIZE  STUDENT  FLOW 

3.  SUPPORT  JUSTIFICATION  FOR  RESOURCE  ALLOCATIONS . 

4.  INCREASE  RESPONSIVENESS  TO  INFORMATION  DEMANDS. 

5.  PROVIDE  UNIQUE  CAPABILITY  TO  TEST  VARIOUS  "MANAGEMENT  STRATEGIES" 
PRIOR  TO  IMPLEMENTATION. 

THE  MAJOR  PORTION  OF  THE  WORK  UNDER  DOTS  IS  BEING  PERFORMED  UNDER 
CONTRACT  BY  THE  INTERNATIONAL  BUSINESS  MACHINES  (IBM)  CORPORATION  IN 
THREE  PHASES.  FHASE  I  WAS  COMPLETED  ON  1  DECEMBER  1973.  THE  FOUR  BASIC 
OBJECTIVES  ACHIEVED  IN  THIS  PHASE  ARE  AS  FOLLOWS: 

1.  A  COMPREHENSIVE  DESCRIPTION  OF  THE  CURRENT  NAVAL  EDUCATION  AND 
TRAINING  SYSTEM,  SHOWING  THE  LINES  OF  INTERACTION  BETWEEN  FUNCTIONS , 
DECISION  POINTS  AND  THE  MANAGEMENT  CONTROLS  ON  THE  SYSTEM- 

2.  STRATEGIC  ASSUMPTIONS  DESCRIBING,  ON  A  PROBABILISTIC  BASIS, 

THE  SOCIAL,  POLITICAL,  ECONOMIC  AND  TECHNOLOGICAL  ENVIRONMENTS  FOR 
THE  1980  DECADE  RELEVANT  TO  NAVY  EDUCATION  AND  TRAINING. 


3.  RECOMMENDATIONS  LEADING  TO,  AND  A  DESCRIPTION  OF,  AN  IDEALIZED 
SYSTEM  IN  TERMS  OF  THE  1980  TIME  FRAME. 

4.  A  LIST  OF  POTENTIAL  OR  EXISTING  COMPUTER-BASED  MODELS  THAT  WILL 
IMPROVE  THE  DECISION-MAKING  PROCESS. 

PHASE  II  OF  THE  PROJECT,  COMPLETED  IN  DECEMBER  1974,  INVOLVED  THE 
SELECTION  OF  THE  MODELS  TO  BE  DEVELOPED,  DESIGN  AND  DEVELOPMENT  OF  Till 
MODELS,  CODING  AND  INITIAL  DEBUGGING,  THE  DEVELOPMENT  OF  DATA  EASES  FC 
MODEL  EXECUTION  AND  TESTING  AND  INITIAL  VALIDATION.  THE  THREE  MODELS 
SELECTED  FROM  A  LIST  CF  21  CANDIDATES  FOR  DEVELOPMENT  IN  THIS  PHASE  ARE 
THE  SYSTEM  CAPABILITIES/ REQUiRt kEnTS  AnD  RESOURCES  MODEL,  TRAINING 
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+  r 


UNCLASSIFIED 


PROCESS  FLOW  MODEL  AND  AN  EDUCATIONAL  TECHNOLOGY  EVALUATION  MODEL. 

EVENTUALLY,  THE  MODELS  SHOULD  COMPLEMENT  THE  NAVY  INTEGRATED  TRAINING 
AND  RESOURCES  ADMINISTRATION  SYSTEM  (NITRAS)  BY  ADDING  A  PREDICTIVE 
CAPABILITY  TO  THE  DATA  BASE. 

PHASE  III  OF  THE  DOTS  PROJECT  IS  PRIMARILY  A  TEST  AND  EVALUATION 
EFFORT.  THE  MODELS  DESIGNED  AND  DEVELOPED  IN  PHASE  JI  HAVE  BEEN 
EXERCISED,  VALIDATED  AND  VERIFIED  AT  THE  FLEET  TRAINING  CENTER,  NORFOLK, 
VIRGINIA  IN  JUNE  1975,  USING  BOTH  REAL-WORLD  AND  SIMULATED  DATA  IN  A 
COMMAND  AND  CONTROL  SETTING.  A  TIME-SHARED,  COMMERCIAL  COMPUTER  TERMINAL 
WAS  INSTALLED  AT  THE  FLEET  TRAINING  CENTER  IN  FEBRUARY  1975. 

A  FORMAL  TEST  AND  EVALUATION  IS  NOW  BEING  CONDUCTED  BY  A  TEAM  LED 
BY  NPRDC  (WASHINGTON  BRANCH.)  ,  TO  DETERMINE  THE  POTENTIAL  FOR  UTILIZING 
THE  MODELS  AT  SIMILAR  TRAINING  ACTIVITIES  AND  THE  DESIRABILITY  OF 
DEVELOPING  RELATED  "PARENT  TYPE"  MODELS  FOR  USE  OF  HIGHER  ORGAANIZATIONAL 
ECHELONS.  OPERATIONAL,  TECHNICAL  AND  FINANCIAL  FEASIBILITIES  WILL  BE 
ASSESSED  IN  THE  TEST  AND  EVALUATION.  FINAL  USER  DOCUMENTATION  WILL  BE 
DELIVERED  AT  THE  COMPLETION  OF  PHASE  III. 

AS  PART  OF  THE  T  AND  E,  NPRDC  HAS  REQUESTED  EACH  USER  TEAM  MEMHEP 
TO  PROVIDE  A  WRITTEN  STATEMENT  OF  COMMAND  POSITION  VIS-A-VIS  THE  DOTS 
PHASE  III  MODELS  AND  DATA  BASE  TO  BE  USED  AS  THE  CAPSTONE  OF  THE  OPERATIONAL 
FEASIBILITY  PORTION  OF  THE  T  AND  E. 

C.  RECOMMENDATIONS/CONCLUSIONS :  RECOMMEND  FOLLOWING  AS  COMTRAPAC  POSITION: 

1.  THE  MODELS  SHOW  PROMISE  AND  SHOULD  BE  FURTHER  DEVELOPED. 

2.  SUCH  FURTHER  DEVELOPMENT  SHOULD  TAKE  PLACE  IN  AN  R  &  D  CONTEXT 
CENTERING  ON  EXPANDING  THE  DOTS  DATA  AND  AFFORDING  FUNCTIONAL  COMMANDERS 
ACCESS  TO  AND  EXPERIENCE  WITH  DOTS  MODELS  AS  A  DECISION  AID. 

3.  A  MAJOR  GOAL  OF  THE  DEVELOPMENT  OF  DOTS  SHOULD  BE  THE  INTEGRATION 
OF  NAVCnrRACOM  AUTOMATED  DATA  SYSTEMS  INTO  COMPLIMENTARY  MODULES  OF  AN 
EFFECTIVE  AND  MULTIFACTED  WHOLE. 

4.  COMTRAPAC  IS  READY  NOW  TO  ESTABLISH  AN  INTERFACE  WITH  THE  MODELS 
AND  TO  CONTRIBUTE  TO  THE  EXPANSION  OF  THE  DATA  BASE  PERSONNEL  RESOURCES 
PERMITTING. 
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TRAINING  PROCESS  FLOW  (TPF)  MODEL 


THE  TPF  MODEL  PROVIDES,  THROUGH  SIMULATION  AND  A  LARGE  STATISTICAL 
DATA  BASE,  THE  MEANS  FOR  ANALYZING  THE  RELATIONSHIP  BETWEEN  STUDENT 
ATTRIBUTES,  TRAINING  DEMANDS ,  AND  COURSE  CHARACTERISTIC  IN  ORDER  TO 
ASCERTAIN  THE  OUTPUT  RATE  OF  THE  TRAINING  SYSTEM  UNDER  CERTAIN  PRESCRIBED 
CONDITIONS.  THE  IMPACT  OF  CHANGES  TO  ANY  OF  THE  VARIABLES  MENTIONED 
AT  THE  COURSE  LEVEL  CAN  BE  ASSESSED  AT  THE  NEXT  HIGHER  ECHELONS,  SUCH 
AS  AT  THE  TRAINING  CENTER  LEVEL.  THE  MODEL  PERMITS  SYSTEM  OPTIMIZATION 
ON  THE  PART  OF  THE  DECISION  MAKER  BY  PERMITTING  HIM  TO  CONSIDER  SUCH 
FACTORS  AS.  THE  MATCH  OF  THE  TRAINING  SYSTEM  CAPABILITY  WITH  JOB  REQUIRE¬ 
MENTS,  SUTDENT  OUTPUT  AS  A  FUNCTION  OF  STUDENT  ATTRIBUTES,  RESOURCE 
USAGE  FOR  GIVEN  OUTPUT  RATES  AND  MIXES.  THE  COSTS  ASSOCIATED  WITH  ANY 
CONCEPTUALIZED  (OR  REAL)  TRAINING  SYSTEM  CAN  BE  OBTAINED  FROM  THE  MODEL. 

SOME  OF  THE  QUESTIONS  THAT  CAN  BE  ASKED  OF  THE  MODEL  ARE  AS  FOLLOWS: 

A.  WHAT  IS  THE  IMPACT  ON  RESOURCES  OF  INCREASING  THE  STUDENT  INPUT 
TO  A  GIVEN  COURSE? 

B.  IF  STUDENT  INPUT  REQUIREMENTS  ARE  LOWERED,  WHAT  WILL  HAPPEN  TO 
STUDENT  ATTRITION?  , 

'  C.  IF  THE  NO-SHOW  RATE  IS  DECREASED  BY  A  CERTAIN  PERCENTAGE,  WHAT 
WILL  HAPPEN  TO  RESOURCE  UTILIZATION? 

D.  WHAT  WILL  BE  THE  EFFECT  OF  CHANGING  THE  WORKING  SCHEDULE? 

E.  HOW  WILL  MODIFICATIONS  TO  PERSONNEL  DETAILING  PROCEDURES  AFFECT 
THE  THROUGHPUT  RATE  AND  PIPELINE  TIME? 

F.  HOW  WILL  THE  THROUGHPUT  RATE  VARY  WITH  CHANGES  IN  INTERNAL 
TESTING  CRITERIA? 

THE  ABOVE  LIST  OF  QUESTIONS  IS  NOT  MEANT  TO  BE  EXHAUSTIVE  BUT  THE 
QUESTIONS  APE  REPRESENTATIVE  OF  THE  KIND  OF  DIALOGUE  POSSIBLE  BETWEEN 
THE  MANAGER  AND  THE  TPF  MODEL. 
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SYSTEM  CAPABILIT1ES/REQUIREMKHTS  AND  RESOURCE  (SCRR)  MODEL 


THE  PURPOSES  OF  THE  SCRR  MODEL  ARE  TO:  (1)  OPTIMISE  THE  MIX  OF 
TRAINING  RESOURCES  (CLASSROOMS,  INSTRUCTORS,  LABORATORIES)  NECESSARY  TO 
ACHIEVE  A  SPECIFIED  OUTPUT  PROFILE  AND  (2)  DETERMINE  THE  MAXIMUM  NUMBER 
OF  COURSE  CONVENINGS  AND  OPTIMAL  MIX  OF  COURSES,  GIVEN  A  SET  OF  RESOURCE 
AND  TIME  CONSTRAINTS.  TO  ACHIEVE  THESE  OBJECTIVES,  IT  WAS  NECESSARY  TO 
DEVELOP  A  TRAINING  RESOURCE  DATA  BASE  WHICH,  AS  A  "BONUS"  FOR  THE  USER, 
ALSO  SERVES  AS  A  STAND-ALONE  INFORMATION  SYSTEM.  THE  MODEL  WILL  PROVIDE 
MANAGERS  WITH  AN  INCREASED  LEVEL  OF  INFORMATION  AND  THE  UNIQUE  CAPABILITY 
OF  ANALYZING  ALTERNATE  STRATEGIES  BEFORE  MAKING  A  DECISION  CONCERNING 
COURSE  CONVENINGS  AND/OR  THE  UTILIZATION  OF  RESOURCES. 

LINEAR  PROGRAMMING  IS  EMPLOYED  IN  ARRIVING  AT  A  DETERMINISTIC 
SOLUTION.  AN  IBM  PROGRAM,  MATHEMATICAL  PROGRAMMING  SYSTEM  EXTENDED 
(MPSX) ,  IS  USED  TO  MAXIMIZE  THE  OBJECTIVE  FUNCTION  REELECTING  STUDENT 
THROUGHPUT.  MPSX  USES  THE  BONDED  VARIABLE/PRODUCT  FORM  OF  THE  INVERSE/ 
REVISED  SIMPLEX  METHOD. 


THE  CONSTRAINING  EQUATIONS,  WHICH  REPRESENT  RESOURCE  LIMITS,  ARE 
KEY  IN  FORMULATING  A  REALISTIC  APPLICATION  OF  THE  MODEL.  SINCE  ONLY  A 
LIMITED  GROUP  OF  INSTRUCTORS  ARE  QUALIFIED  TO  TEACH  EACH  TRAINING  COURSE, 
THE  SUM  TOTAL  OF  THE  INSTRUCTORS'  TIME  AVAILABLE  FOR  CLASSROOM  INSTRUCTION 
LIMITS  THE  FREQUENCY  OF  COURSE  CONVENINGS.  SIMILARLY,  THE  CLASSROOM/ 
LABORATORY  SPACE  IS  LIMITED  AND  THIS  PROVIDES  AN  ADDITIONAL  CONSTRAINT. 
ALSO,  MANY  COURSES  MUST  BE  CONVENED  SOME  MINIMUM  NUMBER  OF  TIMES  ANNUALLY 
TO  FULFILL  A  MINIMUM  TRAINING  REQUIREMENT;  THIS  FURTHER  BOUNDS  THE  SOLUTION 


THE  PREDICTIVE  CAPABILITIES  OF  THE  SCRR  MODEL  ARE  ENHANCED  BY  THE 
ABILITY  TO  MODIFY  THE  TRAINING  RESOURCE  DATA  BASE  AND  OBSERVE  THE  IMPACT 
OF  THE  MODIFICATION.  THIS  CAPABILITY  ENABLES  THE  TRAINING  MANAGER  TO  PLAY 
"WHAT  IF"  GAMES  WITH  THE  MODEL.  SOME  OF  THE  MODIFICATIONS  THAT  COULD  BE 
MADE  TO  THE  DATA  BASE  TO  STUDY  THE  EFFECT  OF  THE  CHANGE  ARE  AS  FOLLOWS: 

A.  COURSES  ADDED  OR  DELETED. 

B.  COURSE  LENGTHS  INCREASED  OR  DECREASED. 

C.  COURSE  FREQUENCY  ALTERED. 

D.  STUDENT/INSTRUCTOR  RATIOS  CHANGED. 

E.  INCREASE  OR  DECREASE  IN  AVAILABILITY  OF  CLASSROOMS,  LABORATORIES 
AND  TRAINING  DEVICES. 


F.  INSTRUCTORS  ADDED  OR  DELETED. 

G.  INSTRUCTOR  QUALIFICATIONS  INCREASED. 

H.  INSTRUCTOR  AVAILABILITY  'HANGED. 
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SYSTEM  CAPABILITIES/REQUIREMKNTS  AND  RESOURCE  (SCRR)  MODEL  (Continued) 


DATA  BASE  ITEM  MODIFICATION,  AS  DESCRIBED  ABOVE,  AFFORD  THE  TRAINING 
MANAGER  THE  OPPORTUNITY  TO  DELETE  OR  VARY  MANY  KEY  PARAMETERS.  THIS 
TYPE  OF  MANIPULATION  COULD  NOT  BE  EASILY  ACCOMPLISHED  MANUALLY.  THE  TRAINING 
RESOURCE  DATA  BASE  AND  THE  OPTIMIZATION  FEATURES  OF  THE  SCRR  MODEL  COUPLE 
TO  OFFER  THE  TRAINING  MANAGER  A  POWERFUL  TOOL  FOR  BOTH  PLANNING  AND 
OPERATION  AND,  AS  A  MINIMUM,  STRUCTURES  THE  RESOURCE  PROBLEM  FOR  FURTHER 
ANALYSIS. 


EDUCATIONAL  TECHNOLOGY  EVALUATION  (ETE)  MODEL 


THE  ETE  MODEL  SIMULATES  THE  FLOW  OF  STUDENTS  THROUGH  A  SELF-PACED, 
INDIVIDUALIZED  LEARNING  SYSTEM.  THE  MODEL  WILL  PROJECT  SYSTEM  OUTPUT, 
AVERAGE  TIME-TO-COMPLETE  AND  INSTRUCTOR  AND  FACILITY  UTILIZATION.  IBM'S 
GENERAL  PURPOSE  SIMULATION  SYSTEM  (GPSS)  IS  USED  IN  THE  MODEL. 

THE  STRATEGY  OF  INDIVIDUALIZED  INSTRUCTION  COMPLICATES  PLANNING  FOR 
SUCH  A  SYSTEM  BECAUSE  STUDENTS  CANNOT  BE  TREATED  AS  GROUPS  OF  INDIVIDUALS 
BEING  TRAINED  TOGETHER.  JUST  TRACKING  STUDENTS  IN  AN  INDIVIDUALIZED 
LEARNING  EVIRONMENT  IS,  IN  ITSELF,  NO  SIMPLE  TASK. 

SOME  OF  THE  QUESTIONS  THAT  THE  MODEL  CAN  ANSWER  ARE  AS  FOLLOWS: 

A.  ARE  THERE  ENOUGH  QULAIFIED  INSTRUCTORS  TO  SUPPORT  AN  INDIVIDUALIZED 
LEARNING  SYSTEM.? 

B.  CAN  ENOUGH  CARRELS  BE  INSTALLED  IN  EXISTING  CLASSROOM  SPACE? 

C.  WHAT  WOULD  BE  THE  IMPACT  ON  COURSE  THROUGHPUT  OF  REMOVING 
ADMINISTRATIVE  DUTIES  FROM  THE  INSTRUCTOR? 

D.  HOW  WILL  THE  AVERAGE  TIME-TO-COMPLETE  THE  SERIES  OF  MODULES 
COMPARE  WITH  TRADITIONAL  COURSE  LENGTH? 

E.  WHAT  IS  THE  PERCENT  UTILIZATION  FOR  INSTRUCTORS  AND  FACILITIES? 

F.  CAN  ADDITIONAL  MODULES  BE  ADDED  TO  THE  COURSE  WITHOUT  BURDENING 
STAFF  OR  FACILITIES? 

G.  IF  COURSES  ARE  CONSOLIDATED,  WHAT  IS  THE  EXPECTED  SYSTEM 
THROUGHPUT? 

AT  PRESENT,  IT  IS  IMPOSSIBLE  TO  ANSWER  THESE  KINDS  OF  QUESTIONS  THROUGH 
CONVENTIONAL  METHODS.  THEREFORE,  THE  ETE  MODEL  IS  A  VITAL  LINK  IN  THE 
TRAINING  SYSTEM  DESIGN  PROCESS. 
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From:  Commander  Training  Command,  U.  S.  Atlantic  Fleet 

To:  Director,  Branch  Office,  Navy  Personnel  Research  and 

Development  Center,  Washington,  DC  20374 

Sub j :  Design  of  Training  Systems  (DOTS);  recommendations 
concerning 

1.  The  Test  and  Evaluation  (T&E)  of  subject  system  conducted 
in  June  of  1975  bad  as  its  objective  the  demonstration  of 
the  applications  of  operations  research  and  management 
science  techniques  in  the  training  community.  The  test,  in 
part,  demonstrated  the  potential  value  of  such  assistance  in 
the  decision-making  progress.  However,  full  utilization  and 
cost  effectiveness  of  such  a  system  was  not  clearly  substan¬ 
tiated;  evaluation  of  the  system  cit  Fleet  Training  Center, 
Norfolk  confirmed  this  opinion.  At  that  command,  the  data 
base  has  been  found  useful  in  providing  internal  management 
information.  The  mathematical  models,  however,  have  net 
been  utilized  by  that  command. 

2.  In  view  of  the  above,  the  following  comments/recommen¬ 
dations  are  submitted: 

a.  Design  of  Training  Systems  (DOTS)  has  potential 
value  for  the  training  community  and  should  be  pursued 
further. 


b.  Development  should  remain  in  the  Research  and 
Development  stage  avid  the  data  base  expanded  sufficiently  to 
permit  evaluations  of  applicability  and  utility  at  higher 
levels  than  previously  tested.  Pursuant  to  this  end,  it  is 
recommended  that  the  test  site  be  relocated  from  Fleet 
Training  Center,  Norfolk,  to  COMTRALANT  Headquarters  to 
facilitate,  experimentation  and  utility  testing  at  the 
functional  command  level.  It  is  also  recommended  that  the 
data  base  bo  expanded  to  include  all  COMTRATjANT  activities 
and  that  access  to  the  system  be  made  available  to  the 


subordinate  activities  for  both  training  and  evaluation. 
Central  siting,  for  the  present,  appears  to  bo  the  only 
cost-effective  manner  in  which  to  proceed. 


F-13 


c.  While  the  value  of  DOTS  as  an  independent  r-ystem  is 
understood ,  emphasis  should  bo  placed  upon  its  incorporation 
into  the  N  AV  r,  D  T  RAC  CM  data  system  with  free  exchange  of  ideas 
and  informal ion  as  a  final  goal  on  an  open  reciprocal  basis. 

d.  COMTl’J J .7iU j’  is  prepared  tc  participate  in  further 
experimentation  and  development  of  this  system  which,  thus 
far,  has  demonstrated  to  a  significant  extent  its  potential 
value  to  the  training  community. 


Code  015 

JSOO 

6  AUG  1975. 


-  From:  Chief  off  Naval  Technical  Training 
To:  Chief  of  Naval  Education  and  Training 

*  •  i 

Subj:  Design  of  Training  Systens  (DOTS)  '  . 

4  ' 

•  ' 

Ref:  (a)  CNET  ltr  Codo  01  of  14  May  1975 

X.  Uy  reference  (a),  the  Chiof  of  Naval  education  nnd  Training  roe, nested 
that  the  Chief  of  Naval  Technical  Training  submit  an  evaluation  of  the 
DOTS  following  the  completion,  of  the  operational  T  5  Ji  (Test  ami  Evaluation) 
at  Fleet  Training  Center  Norfolk.  ... 

Tho  Chief  of  Naval  Technical  Training  has  weighed  carefully  the 
purposes,  operational  results,  previous  investment,  and  potential  of  the 
DOTS  models.  The  following  conclusions  apply: 

tv.  The  T  £  E  at  Fleet  Training  Center  Norfolk  did  not  show  the 
"substantiated  proof  of  operational  desirability,  feasibility  and 
economic  acceptability"  set  forth  as  criteria  in  reference  (a)  to 
warrant  tasking  within  tho  EDTRACCiJ  for  sponsorship,  management,  development 
and  implementation.  • 


h-  The  evaluation  at  Fleet  Training  Center  Norfolk  supports  tho 
earlier  cone}  ns 5 o\i  of.  the  Chief  of  Naval  Technical  Training  that  the 
models  h-'vo  been  primarily  designed  to  he'i p  that  level  of  management  ' 
Icvisi;  requiring  assistance  in  decision-making.  This  general  conclusion 
was  borne  out  by  tho  Director  of  Training,  Fleet  Training  Center, 
Norfolk,  who  indicated  that  use  of  the  models  for  "what  if”  questions 
would  ho  rave,  and  that:  his  most  frequent  use  was  simply  as  n  report- 
typo  vcfeieuco  (not  the  avowed  province  of  the  models).  Although  it  is 
qijp.rront  that  tho  System  Capabilit ics/Ucquiromonts  and  Uosourc.es  (SCUk) 
Model  and  tin:  Training  I'recess  J:.lr.v:  (TI’F)  Model  do  have  curtain  problem 
cpi j to  1 5. v.at i on  capabilities  and  arc  wojkablc  at  tho  ccboolhouso  level, 
it  )ins  not  been  demons t rated  that  these  models  would  be  extensively  used 
or  cost  effective.  It  was.  in  fact  noted  t  hr: t  an  creel  lent  and  obvious 


oupo;:  ■.  ual  i  y  vo  uso  the. 
(allocation  of  a  staff 
Training  Center  during 


models  to  nelp  solve  a  current  practical  problem 
aj.iowr.nce  decrement)  was  not  taken  by  tho  Fleet 
tho  time  of  the  T  (•  E. 
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c.  Two  specific  technical  deficiencies  of  the  SCRR  and  TJ’F  pjodols 
were  evident.  These  two  models,  both  concerned  with  resultant  impacts 
on  training  ar-  n  function  of  resource  and  demand  variations  are  not 
interactive  t  They  should  be.  This  night  well  bo  a  rolativoly  simple 
programming  change  to  effect,  but  the  point  is,  as  presently  structured 
they  arc  .not.  Secondly,  and  more  importantly,  the  models  are, not 
.Itcrati  While  any  one  or  more  of  the  variables  for  a  particular 
problem  may  be  run  and  an  optimal  solution  for  that  problem  provided, 
the  model (s)  will  not  accommodate  a  reiterative  matrix  of  tho  same 
variable  and  automatically  optimize  within  the  vango  of  that  matrix. 

For  example,  if  wo  wanted  to  sec  what  impact  on  X  %  decrease  in  feci 15 vies 
would  have  on  training,  the  model  would  provide  an  answer;  but  if  wo 
wanted  to  know  what  maximum  decrease  in -facilities  we  could  accept 
within  a  range  of  X  to  Y  v  and  still  continue  training,  we  would  have  to 
run  the  model  for  each  and  every  number  within  that  range.  Even  more 
significant  is  the  fact  that  only  by  independent,  single  variable  - 
change  runs  can  the  limiting  or  key  variable  in  any  problem  bo  identified. 
'Therefore  multiple  runs,  separately  initiated,  must  be  made  in  which 
each  variable  is  independently  changed  and  whereupon  the.  solution  cqn 
therefore  bo  accurately  related  to  that  specific  variable. 

d.  In  term  of  functional -level  use,  tho  SCRR  and  TP?  models  ns 
presently  configured  will  not  substantially  assist  the  Chief  of  Naval 
technical  Training  in  either  predicting/allocating  resources  or  in 
assessing  impact  on  student:  output  of  resource  changes  appreciably 
beyond  the  capability  which  now  exists  or  is  envisioned  from  NJTJIAS, 
SKORSTAMl’S,  RMS  MCC  and  other,  systems.  However,  it  is  probable  that 
higher-echelon  variations  of  the  SCRR  and  TPF  could  be.  developed. 

c.'  1't  is  considered  that  continuance  of  upward  development  aimed  r.v. 
eventually  satisfying  the  above  requirements  is  not  justifiable  solely 
by  the  fact  that  such  development  would  be  paid  from  R  S  D  rather  than 
operating  funds.  This  is  particularly  true  since  it  is  anticipated  that 
sufficient  decision-making  assistance  will  bo  available  from  other 
systems,  end  further  redundancy  cf  computer  programs  and  products  should 
bo  prevented. 
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f.  It  is  imperative  that  early  efforts  bo  made  to  define  and 
coordinate*  those  portions  of  other  research  and  systevis  development 
efforts  which  appear  to  have  objectives  that  are  redundant  to  DO'lh  and 
to  each  other.  Lxamplcs  of  these  include  STUFFS,  SHOUSTAMi’S,  CFN7RA, 

N1TRAS,  MATH',  MAPMIS,  UACC ,  etc.  Tins  potential  overlap  will  increase 
villi  the  vertical  evolution  of  SC11R  and  IFF  to  higher  management  levels.  , 

f.  • 

g.  T/ic  F.ducntional  Technology  Evaluation  (FTP.)  Model  is,  if  nothing 
ol‘;o.  Misnamed.  It  does  not  evaluate  educational  technology.  If  doe? 
simulate  tlie  flow  of  students  through  a  self-paced,  individualized 
training  system  and  therefore  provider,  n  neons  of  analyzing  pipeline. 

Ibis  analytical  capability  is  however,  more  predictive  than  prescriptive 
in  that  while  it  provides  rapid  analysis  of  cause  and  effect  phenomena 
impacting  upon  a  given  system,  it  cannot  in  itself  prescribe  the  one 
optimal  configuration.  Tliis  model  also  la  eh--  an  iterative  capability. 

i».  In  .Mr.nxary ,  tho  objective  of  The  DOTS  Project  is  “to  improve  the 
manage-.;. ent  of  the  Navy’s  Training  system  b>  ‘providing  an  expanded 
decision-making  capability  for  all  levels  of  Management. . .to  bo  used  by 
training  Managers  in  dealing  with  the  various  social  and  economic, 
factors  and  with  the  technological  advances  that  will  impinge  on  Navy 
training,  through  the  IPSO  decade."  Although  the  SCRU,  Tl’F  and  LIT 
models  will  fit  within  this  broad  objective  statement ,  their  utility  has 
not  been  established.  The  value  of  the  DOTS  models  at.  the  schoolhodsc- 
level  to  assist  or  replace  the  on-site  judgment  of  school  managers 
remains  quest' enable.  Also,  considerable  evolution  will  bo  required  to 
meet  the  objective  “for  all  levels  of  Management,"  This  evolution  would 
<“»11  for  an  extended  period  of  R  C,  D,  mid  the  results  could  he  increasingly 
redundant  with  other  pod els  or  systems  unless  some  of  the  latter  ore 
eliminated. 

4.  Recommendations: 

n.  That  the  POTS  project  be  terminated  at  the  expiration  of  the 
current  contract. 

h.  That  upon  completion  of  the  current  contract,  the  HOTS  models, 
supporting  software  and  documentation  be  provided  to  the  MKTITA  lor 
investigation  as  to  their  appl icnbil i ty  cud  utility  within  on tent 
hiriUACOM  ADI*  systems . 
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COMMENTS  ON  THE  ANALYSIS  OF  STUDENT  FAILURE  RATES 


Since  "failure"  in  this  study  is  a  zero-one  type  variable, 
the  analysis  of  possible  relationships  between  failure  or 
the  failure  rate  and  other  student,  class,  or  course 
characteristics  raises  some  special  problems.  These  can 
best  be  considered  by  first  making  a  clear  distinction 
between  the  "sampling  unit"  and  the  "prediction  unit." 

In  most  regression  analysis  these  are  the  same;  here  they 
may  not  be  and,  in  fact,  in  the  present  analysis  are  not. 

By  "sampling  unit"  we  mean  the  element  from  which  data  is 
originally  collected.  In  this  case  it  can  either  be  the 
individual  student  or  the  group  (class  or  course) .  in  a 
similar  way,  the  "prediction  unit"  can  be  either  the  in¬ 
dividual  or  the  group;  we  can  either  attempt  to  predict 
success  or  failure  for  the  first  (a  zero-one  variable)  or 
the  number  or  proportion  of  failures  for  the  latter  (the 
failure  rate) . 

Using  these  cr iter  id  we  can  distinguish  three  different 
cases : 1 

Case  I  Sampling  and  prediction  unit  -  the 
individual  student. 

Case  II  Sampling  and  prediction  unit  -  the  group. 

Case  in  Sampling  unit  -  the  individual  student; 
prediction  unit  -  the  group. 

Case  I  Sampling  and  prediction  unit  -  the  individual 
student . 


^In  general  we  can  aggregate  for  purposes  of  prediction 
but  not  disaggregate.  Thus,  a  fourth  case,  group 
sampling  and  individual  prediction  is  not  considered. 

It  would,  however,  be  possible  to  expand  Case  III  to 
cover  groups  of  different  sizes.  For  example,  we  could 
use  the  class  as  sampling  unit  but  predict  failure  rates 
for  the  course  or  school. 


This  case  is  the  one  referred  to  in  the  literature 
as  Discriminant  Analysis.2  We  assume  that  the  population 
of  students  can  be  dichotomized  into  two  disjoint  sub¬ 
populations,  successes  and  failures.  For  each  sub¬ 
population  it  is  assumed  that  a  multivariate  distribution 
of  predictor  variables  exists  and  that  these  distributions 
differ  only  in  their  vector  of  expected  values.  In  the 
usual  case  it  is  assumed  that  the  distributions  are  multi¬ 
variate  Normal  with  the  same  var iance-convar iancc  matrix. 

Discriminant  Analysis  then  consists  of  an  attempt  to 
find  a  linear  "discriminant  function"  which  creates  two 
half-spaces  associated  with  successes  and  failures  in  such 
a  way  that  it  minimizes  the  probability  of  misclass if ica- 
tion.  in  this  case  the  prediction  is  simply  "success"  or 
"failure"  for  the  individual  student.  It  should  be  noted 
that  this  approach  assumes  "failures"  are  qualitatively 
different  and  not  just  those  students  below  a  given  quanti¬ 
tative  cutoff. 

Case  II  Sampling  and  prediction  unit  -  the  group. 

The  dependent  variable  in  Case  II  is  either  the  number 
or  the  proportion  of  students  in  the  group  who  fail.  This 
in  some  cases  may  be  a  Bmomially  distributed  variable 
but  the  possibility  of  a  more  complex  underlying  proba¬ 
bility  process  cannot  be  completely  ignored.  On  the  other 
hand,  if  the  group  size  is  larger,  the  distribution  of  the 
proportion  of  failures  (the  failure  rate)  will  in  many 
cases  be  approximately  Normal.^ 


g 

4For  example,  see  Guilford,  J.  D.  Psychometric  Methods, 
McGraw-Hill,  1954,  P  364-365. 

^The  critical  Assumption  would  seem  to  be  whether  or  not 
individuals  in  the  group  succeed  or  fail  independently. 
Intra-sample  correltion  can  have  a  critical  impact  on 
the  validity  of  the  Central  Limit  Thecrm. 
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The  independent  variables  will  be  characteristics 
of  the  group  rather  than  of  the  individual.  This  does 
not  preclude  the  use  of  group  collective  measures  such 
as  mean  QCT  or  its  standard  deviation.  For  Case  II,  the 
most  critical  problem  relates  to  the  assumption  of 
homoskedasticity  necessary  for  the  usual  use  of  regression 
analysis.  If  we  assume  that  the  dependent  variable  follows 
the  Binomial  distribution,  it  follows  that  both  its 
variance  and  expected  value  will  depend  on  the  prediction 
variables.  In  this  case  it  is  possible  to  transform  the 
independent  variable  so  that  the  assumptions  hold.4  In 
the  more  general  case  the  transformation  may  be  more 
difficult  to  develop. 

Case  III  Sampling  unit  -  the  individual  student?  pre¬ 
diction  unit  -  the  group 

For  the  mixed  case  represented  here,  a  number  of  very 
difficult  problems  arise.  These  can  best  be  illustrated 
by  considering  a  simple  model  with  one  predictor  variable. 
Figure  1  illustrates  the  nature  of  the  data  for  individual 
students.  It  can  be  seen  that  most  of  the  assumptions 
associated  with  regression  analysis  are  either  incorrect 
or  untestable.  Linearity  of  the  underlying  model  is  un¬ 
likely  since  the  relationship  will  be  asymtotic  at  both 
zero  and  one.  The  errors  are  not  normally  distributed; 
for  a  given  value  of  x  they  may  be  binomial.  It  follows 
that,  as  in  case  II,  the  variance  is  a  function  of  x. 

While  the  F  test  is  known  to  be  reasonably  robust,  it  is 
not  at  all  clear  that  it  can  be  applied  here. 

Additional  problems  arise  in  using  this  model  to  pre¬ 
dict  failure  rates  for  groups.  First,  as  noted  under 
Case  II,  there  may  be  intraclass  correlation  so  that  the 
failure  rate  for  the  class  will  be  different  than  the 
average  failure  probability  of  the  members.  More  critical 
is  the  use  of  the  group  average  of  the  x  variable  as  an 
input  to  the  model,  it  is  at  this  point  that  the  assump¬ 
tion  of  linearity  in  the  model  may  become  a  major  problem. 


4See  material  on  Probit  Analysis  in  J.  D.  Finney,  Probit 
Analysis  (2nd  ed.),  Cambridge  University  Press  (1962). 
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As  a  simple  example,  assume  x  is  perfectly  corre¬ 
lated  with  the  course  grade  so  that  there  is  some 
critical  value  of  x  (say  x^)  and  all  students  below  xc 
fail.  A  possible  distribution  of  x  is  shown  in  Figure 
2a.  It  can  be  seen  that  increasing  x  (shifting  the 
distribution  to  the  right)  will  decrease  the  proportion 
of  failures  in  a  non-linear  way.  In  fact,  if  the  dis¬ 
tribution  of  x  is  normal,  failure  rates  around  11%,  as 
used  in  the  report,  are  close  to  the  point  of  maximum 
curvature  of  the  function. 

Because  of  this  non-linearity,  x  does  not  adequately 
summarize  the  distribution  of  x's  in  the  group.  For 
example,  in  Figure  2b  equal  numbers  of  additional  students 
are  added  to  the  group  above  and  below  x.  x  is  not 
changed  but  the  failure  rate  is  substantially  increased. 

Summary 

The  problems  associated  with  estimating  group  failure 
rates  from  data  on  individual  students  suggests  that  a 
more  practical  approach  might  be  to  work  directly  from 
group  failure  rates  for  classes.  (That  is  to  use  the  clas 
as  the  sampling  unit.)  Even  here  some  transformation  such 
as  the  "probit"  will  be  necessary  to  remove  hetroskedasti- 
city.  Problems  of  aggregating  class  failure  rates  into 
course  or  school  aggregates  will  still  cause  some  problems 
but  appear  much  more  tractable  than  the  approach  used 
previously. 
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